Updated Stan Syntax Highlighting in VS Code

General
#1

I don’t know if many of you use VS Code but the extension was very outdated.

I’ve made a pull request at the Stan VS Code Extension GitHub repository that was approved and the new version is now available in VS Code Extension Marketplace.

I’ve inserted a lot of knew functions and language features to make it up-to-date with Stan version 2.26. (I might have missed some functionalities).

If you use VS Code try it out and, if you can, please contribute with better functionality and syntax highlighting.

Here is the full details on the new features and fixes:

Changes
  • QR decomposition
    • Added qr_Q and qr_R to deprecated functions.
    • added qr_thin_Q and qr_thin_R to functions.The functions qr_thin_Q and qr_thin_R implement the thin QR decomposition, which is to be preferred to the fat QR decomposition that would be obtained by using qr_Q and qr_R , as the latter would more easily run out of memory … (Section 1.2 Stan’s User Guide)
  • Moved functions to deprecated functions (Section 10 of Stan Function Reference)
    • integrate_ode_rk45
    • integrate_ode_bdf
    • integrate_ode_adams
  • New unnormalized distributions (Stan v 2.25)
    • Changed all *_lpdf to also accept the new *_lupdf (unnormalized log PDF) – _l[u]?pdf (Section of Discrete Distributions in Stan Functions Reference).
    • Changed all *_lpmf to also accept the new *_lupmf (unnormalized log PDF) – _l[u]?pmf (Section of Continuous Distributions in Stan Functions Reference).
  • New functions
    • hmm_marginal function (Stan v 2.24)
    • reduce_sum and reduce_sum_static functions (Stan v 2.23)
    • map_rect function (Stan v 2.18)
    • integrate_1d function (Stan v 2.18)
    • algebra_solver and algebra_solver_newton functions (Stan v 2.24)
  • New GLM more efficient distribution functions
    • Added the new discrete distribution family bernoulli_logit_glm Section 12.3 Stan Functions Reference) – bernoulli_logit_glm and bernoulli_logit_glm_l[u]?pmf .
      • Added the new discrete distribution family ordered_logistic_glm Section 13.9 Stan Functions Reference) – ordered_logistic_glm and ordered_logistic_glm_l[u]?pmf .
    • Added the new discrete distribution family neg_binomial_2_log_glm Section 14.4 Stan Functions Reference) – neg_binomial_2_log_glm and neg_binomial_2_log_glm_l[u]?pmf .
    • Added the new discrete distribution family poisson_log_glm Section 14.7 Stan Functions Reference) – poisson_log_glm_lupmf and poisson_log_glm_l[u]?pmf .
    • Added the new continuous distribution family normal_id_glm (Section 16.2 Stan Functions Reference) – normal_id_glm and normal_id_glm_l[u]?pdf .
  • New distributions
    • discrete_range (Section 13.7 Stan Functions Reference) – discrete_range and [ discrete_range_cdf|discrete_range_lccdf|discrete_range_lcdf|discrete_range_l[u]?pmf|discrete_range_rng ]
    • ordered_probit (Section 13.10 Stan Functions Reference) – ordered_probit and [ ordered_probit_l[u]?pdf|ordered_probit_rng ]
    • multinomial_logit (Section 15.2 Stan Functions Reference) – multinomial_logit and [ multinomial_logit_l[u]?pdf|multinomial_logit_rng ]
    • beta_proportion (Section 19.2 Stan Functions Reference) – beta_proportion and [ beta_proportionbeta_proportion_lccdf|beta_proportion_lcdf|beta_proportion_l[u]?pdf|beta_proportion_rng ])
10 Likes
#2

Fantastic @storopoli ! Love this. I use VSCode for almost all of my development work so this is super useful,

1 Like
#3

@rok_cesnovar hold your horses. I’m working on a new pull request that will add snippets 😀
temp

3 Likes
#4

Very nice!

@storopoli would you know if there is a size limit to these plugins?

Given that this is all just javascript we could easily incorporate the stanc3.js in this (we build stanc3 also as a 1.7MB JS file). We could do syntax checks, show error messages in place, and so on. I have a simple syntax check working that I worked on some time ago but never had the time to go any further with that. Its really easy to use. See here and here

#5

@rok_cesnovar, my knowledge in JavaScript is NULL. I mainly code in C++, R, Python and Julia. From what I understood we need to create a Language Server for Stan. See here.

Language Server is a special kind of Visual Studio Code extension that powers the editing experience for many programming languages. With Language Servers, you can implement autocomplete, error-checking (diagnostics), jump-to-definition, and many other language features supported in VS Code.

I don’t think there is a size limit to the extensions, if there is 1.7MB won’t be a problem. I also think that converting stanc.js to .ts and creating the Language Server would not be so difficult.

We could talk with the developer of the VSCode extension Ivan Bocharov if he would agree to handover the repository to stan-dev then we can begin to plan and do further improvements. From what I saw seeing his CV and GitHub contributions, Ivan is a PhD candidate at Eindhoven University of Technology and is mainly working/developing Julia packages for his Eindhoven lab/supervisors.

1 Like
#6

Hi all,

I am the original developer of Stan syntax highlighting package for VSCode. I have developed the package a few years ago only with myself and my peers in mind. I am glad that the package has caught some attention and want to thank @storopoli for his recent contributions. With his PRs the package now supports the latest version of Stan, has code snippets and supports folding.

As I am finishing my PhD soon, I will have more time for my own interests and would very much like to be involved in Stan development. I don’t have any objections to transferring the ownership to stan-dev, but I would love to participate in the future evolution of the package.

As for LSP implementation, we can definitely have a look at what is currently possible/exposed. I remember checking stanc3 repo about a year ago to see if there’s any development on LSP support and finding this issue that seemed relevant at the time. Is there a coordinated effort inside Stan organization to work on LSP support?

@rok_cesnovar I had a look at the code you have shared. It seems that any change would require a recompilation to highlight possible errors etc. Did I get this correctly? Ideally, of course, the server would be using some diagnostic info on-the-fly, but I understand that it might be problematic. That’s why I was asking whether there’s some work internally on exposing APIs for e.g. syntax errors pre-compilation. At the same time, I don’t think that embedding stanc3 would be a problem size-wise.

2 Likes
#7

Welcome @ivan-bocharov!

Yes, right now that would mean recompiling. Though recompiling is basically instant, we should think about a LSP, I agree.

I was thinking more if we could add functionalities like doing an actual syntax check, running the canonicalizer, doing a pedantic check, etc. These are all available with stanc3 via different flags.
But dont get me wrong, this add-on is fantastic as is already.

#8

I played a bit more with your demo and I love it. I especially like autoformatting and syntax checks via compilation for the purpose of the extension. I will have a look at what has changed in the VSCode LSP APIs since last time I checked and will try to integrate some of stanc3 features into the package.

1 Like
#9

Great! Looking forward to any updates.

Oh and also good luck with the final PhD steps!

1 Like
#11

@ivan-bocharov good luck on your final PhD steps. And thank you for the kindness in the PRs, and for the great extension.

Guys, let me know how I can help in further developments.

2 Likes
#12

Thanks everyone! There’s still quite some work to be done and I probably should’ve used “leaving TU/e” instead of “finishing” :)

I’ll keep posting updates in this thread unless we discover something that would warrant a new topic.

1 Like
#13

@ivan-bocharov @storopoli If any of you two spare a few moments and want to mae a quick write up with some images/gifs I would be very happy to post this on our Stan blog and tweet it out from our Stan Twiiter account to get some more exposure it deserves. It does not have to be lengthy.

No pressure though.

1 Like
#14

@rok_cesnovar Thanks! I’ll write a short post by the end of the week!

1 Like
#15

Has there been any update on the initial stages of the effort to get Stan a language server? It would go a long way towards easing some of the challenges of writing Stan code, especially for newer users. A decent language server implementation would also allow for a more universal experience across a whole bunch of tools/editors – VS Code, Jupyter, vim/emacs, etc…

1 Like
#16

No updates from my part. The main problem is that the LS I think is based on JavaScript which I don’t know anything…

1 Like
#17

My understanding is a LS can be implemented in any language. The OCaml language server is implemented in OCaml. I believe it’s reasonable to re-use a lot of the existing functionality in stanc3 as part of this – see, for example, this python LS which implements the protocol but stubs out much of the functionality to existing packages like Jedi or Rope.

If someone would like to start a project for this I would be happy to contribute – my only request is we don’t do it in Javascript

#18

The language server extension guide that VS Code puts forward does a good job of explaining the process:

  1. The language client would have to be in Javascript/Typescript, which looks like it probably does nothing more than take data about the files being edited and passing them to the language server via JSON over the language server protocol.
  2. The language server which is the backend process doing the actual code analysis and returning things like autocomplete/linting/error checking/etc, can be written in any language.

Not sure what would be the best approach for the Stan community, but I’d imagine something in the Python or Julia space would be the lightest lift for writing the LSP.

2 Likes
#19

If the idea is to simply call stanc3 and let it handle the errors, etc I think Python is a good lightweight choice. I’m not sure if that will get very far in the end however, because I think a lot of the LSP features depend on fail-slow parsing which tries to parse even after encountering an error, which stanc3 doesn’t do. The library we use in OCaml for parsing supports it, so we could copy the grammar directly and set up a fault tolerant parser, but then you’re tied to OCaml as the implementation language

1 Like
#20

Definitely would restrict some of the on-the-fly error testing, but I imagine the autocomplete or built-in documentation/function signature features would be really valuable on their own.

Perhaps an initial implementation that focuses on documentation-based features would be easier than trying to get stanc3 to perform fail-slow parsing.

3 Likes
#21

@ivan-bocharov since PR Support .stanfunctions as an extension for files containing only function definitions/declarations by WardBrian · Pull Request #1022 · stan-dev/stanc3 · GitHub there is another valid stan file ending .stanfunctions. Would it be possible to update the stan-vscode to recognize file endings with .stanfunctions?

5 Likes