I’ve inserted a lot of knew functions and language features to make it up-to-date with Stan version 2.26. (I might have missed some functionalities).
If you use VS Code try it out and, if you can, please contribute with better functionality and syntax highlighting.
Here is the full details on the new features and fixes:
Changes
QR decomposition
Added qr_Q and qr_R to deprecated functions.
added qr_thin_Q and qr_thin_R to functions.The functions qr_thin_Q and qr_thin_R implement the thin QR decomposition, which is to be preferred to the fat QR decomposition that would be obtained by using qr_Q and qr_R , as the latter would more easily run out of memory … (Section 1.2 Stan’s User Guide)
Moved functions to deprecated functions (Section 10 of Stan Function Reference)
integrate_ode_rk45
integrate_ode_bdf
integrate_ode_adams
New unnormalized distributions (Stan v 2.25)
Changed all *_lpdf to also accept the new *_lupdf (unnormalized log PDF) – _l[u]?pdf (Section of Discrete Distributions in Stan Functions Reference).
Changed all *_lpmf to also accept the new *_lupmf (unnormalized log PDF) – _l[u]?pmf (Section of Continuous Distributions in Stan Functions Reference).
New functions
hmm_marginal function (Stan v 2.24)
reduce_sum and reduce_sum_static functions (Stan v 2.23)
map_rect function (Stan v 2.18)
integrate_1d function (Stan v 2.18)
algebra_solver and algebra_solver_newton functions (Stan v 2.24)
New GLM more efficient distribution functions
Added the new discrete distribution family bernoulli_logit_glm Section 12.3 Stan Functions Reference) – bernoulli_logit_glm and bernoulli_logit_glm_l[u]?pmf .
Added the new discrete distribution family ordered_logistic_glm Section 13.9 Stan Functions Reference) – ordered_logistic_glm and ordered_logistic_glm_l[u]?pmf .
Added the new discrete distribution family neg_binomial_2_log_glm Section 14.4 Stan Functions Reference) – neg_binomial_2_log_glm and neg_binomial_2_log_glm_l[u]?pmf .
Added the new discrete distribution family poisson_log_glm Section 14.7 Stan Functions Reference) – poisson_log_glm_lupmf and poisson_log_glm_l[u]?pmf .
Added the new continuous distribution family normal_id_glm (Section 16.2 Stan Functions Reference) – normal_id_glm and normal_id_glm_l[u]?pdf .
New distributions
discrete_range (Section 13.7 Stan Functions Reference) – discrete_range and [ discrete_range_cdf|discrete_range_lccdf|discrete_range_lcdf|discrete_range_l[u]?pmf|discrete_range_rng ]
ordered_probit (Section 13.10 Stan Functions Reference) – ordered_probit and [ ordered_probit_l[u]?pdf|ordered_probit_rng ]
multinomial_logit (Section 15.2 Stan Functions Reference) – multinomial_logit and [ multinomial_logit_l[u]?pdf|multinomial_logit_rng ]
beta_proportion (Section 19.2 Stan Functions Reference) – beta_proportion and [ beta_proportionbeta_proportion_lccdf|beta_proportion_lcdf|beta_proportion_l[u]?pdf|beta_proportion_rng ])
@storopoli would you know if there is a size limit to these plugins?
Given that this is all just javascript we could easily incorporate the stanc3.js in this (we build stanc3 also as a 1.7MB JS file). We could do syntax checks, show error messages in place, and so on. I have a simple syntax check working that I worked on some time ago but never had the time to go any further with that. Its really easy to use. See here and here
@rok_cesnovar, my knowledge in JavaScript is NULL. I mainly code in C++, R, Python and Julia. From what I understood we need to create a Language Server for Stan. See here.
Language Server is a special kind of Visual Studio Code extension that powers the editing experience for many programming languages. With Language Servers, you can implement autocomplete, error-checking (diagnostics), jump-to-definition, and many other language features supported in VS Code.
I don’t think there is a size limit to the extensions, if there is 1.7MB won’t be a problem. I also think that converting stanc.js to .ts and creating the Language Server would not be so difficult.
We could talk with the developer of the VSCode extension Ivan Bocharov if he would agree to handover the repository to stan-dev then we can begin to plan and do further improvements. From what I saw seeing his CV and GitHub contributions, Ivan is a PhD candidate at Eindhoven University of Technology and is mainly working/developing Julia packages for his Eindhoven lab/supervisors.
I am the original developer of Stan syntax highlighting package for VSCode. I have developed the package a few years ago only with myself and my peers in mind. I am glad that the package has caught some attention and want to thank @storopoli for his recent contributions. With his PRs the package now supports the latest version of Stan, has code snippets and supports folding.
As I am finishing my PhD soon, I will have more time for my own interests and would very much like to be involved in Stan development. I don’t have any objections to transferring the ownership to stan-dev, but I would love to participate in the future evolution of the package.
As for LSP implementation, we can definitely have a look at what is currently possible/exposed. I remember checking stanc3 repo about a year ago to see if there’s any development on LSP support and finding this issue that seemed relevant at the time. Is there a coordinated effort inside Stan organization to work on LSP support?
@rok_cesnovar I had a look at the code you have shared. It seems that any change would require a recompilation to highlight possible errors etc. Did I get this correctly? Ideally, of course, the server would be using some diagnostic info on-the-fly, but I understand that it might be problematic. That’s why I was asking whether there’s some work internally on exposing APIs for e.g. syntax errors pre-compilation. At the same time, I don’t think that embedding stanc3 would be a problem size-wise.
Yes, right now that would mean recompiling. Though recompiling is basically instant, we should think about a LSP, I agree.
I was thinking more if we could add functionalities like doing an actual syntax check, running the canonicalizer, doing a pedantic check, etc. These are all available with stanc3 via different flags.
But dont get me wrong, this add-on is fantastic as is already.
I played a bit more with your demo and I love it. I especially like autoformatting and syntax checks via compilation for the purpose of the extension. I will have a look at what has changed in the VSCode LSP APIs since last time I checked and will try to integrate some of stanc3 features into the package.
@ivan-bocharov@storopoli If any of you two spare a few moments and want to mae a quick write up with some images/gifs I would be very happy to post this on our Stan blog and tweet it out from our Stan Twiiter account to get some more exposure it deserves. It does not have to be lengthy.
Has there been any update on the initial stages of the effort to get Stan a language server? It would go a long way towards easing some of the challenges of writing Stan code, especially for newer users. A decent language server implementation would also allow for a more universal experience across a whole bunch of tools/editors – VS Code, Jupyter, vim/emacs, etc…
My understanding is a LS can be implemented in any language. The OCaml language server is implemented in OCaml. I believe it’s reasonable to re-use a lot of the existing functionality in stanc3 as part of this – see, for example, this python LS which implements the protocol but stubs out much of the functionality to existing packages like Jedi or Rope.
If someone would like to start a project for this I would be happy to contribute – my only request is we don’t do it in Javascript
The language client would have to be in Javascript/Typescript, which looks like it probably does nothing more than take data about the files being edited and passing them to the language server via JSON over the language server protocol.
The language server which is the backend process doing the actual code analysis and returning things like autocomplete/linting/error checking/etc, can be written in any language.
Not sure what would be the best approach for the Stan community, but I’d imagine something in the Python or Julia space would be the lightest lift for writing the LSP.
If the idea is to simply call stanc3 and let it handle the errors, etc I think Python is a good lightweight choice. I’m not sure if that will get very far in the end however, because I think a lot of the LSP features depend on fail-slow parsing which tries to parse even after encountering an error, which stanc3 doesn’t do. The library we use in OCaml for parsing supports it, so we could copy the grammar directly and set up a fault tolerant parser, but then you’re tied to OCaml as the implementation language
Definitely would restrict some of the on-the-fly error testing, but I imagine the autocomplete or built-in documentation/function signature features would be really valuable on their own.
Perhaps an initial implementation that focuses on documentation-based features would be easier than trying to get stanc3 to perform fail-slow parsing.
