My experiment on webstan/cloudstan

#1

I know the idea of a webstan/cloudstan has been brewing around here for quite some time. I was running some performance tests and decided to experiment with cloudstan a bit. I made a live demo that runs on a free AWS instance so its understandably slow.

Underneath it uses httpstan (thanks @ariddell and everyone else working on that, it was a breeze to make a new interface work).

You can check it out here: https://cloudstan-experiment.herokuapp.com/

Logging in is mandatory in order to create and run a model. If you dont want to register, you can use dummy data (the email has to be in valid form, test@test.com is fine, no emails will be sent). Once you login you can create a new model, compile it, supply the data and get the results in table and chart form (some basic charts).

The editor has syntax highlighting, but I havent yet added all the keywords, just some quick ones. The fit summary doesnt include n_eff, Rhat and se_mean as those require a bit more calculation and httpstan doesnt calculate them.

If for some reason the AWS instance would terminate, here are some images of how it looks like.

cloudstan1
cloudstan1.png1829Ă—946 67.3 KB
2
2.png1822Ă—947 58.6 KB
3
3.png1815Ă—937 90.5 KB
4
4.png1215Ă—858 44.3 KB
5
5.png1153Ă—844 173 KB

The charts display 1 parameter by default, other can be added by clicking their name in the charts. For the histogram you can select the parameter from the dropdown.

Would something like this be of any use to anyone? I imagine it could be useful for beginners. Any comments or discussion is welcome.

The main issue is of course the computing resources, as the free instance will probably be limiting.
EDIT: I havent tested it on mobile, not sure it works there.

2 Likes
#2

Hi, quick way to get ess and rhat could be done by wrapping arviz?

1 Like
#3

Rok,
That is really nice and above all simple. Sorry to immediately request a feature but access to the errors would help debug models. My model didn’t compile but works locally but not worth worrying about.

I am teaching/developing Stan for Programmers, this is a very nice interface that may work better than RStudio. Issue for classes is scaling to more instances as 40 people compile their own Stan program at the same time.

Do you need resources to keep moving this along? SGB has funds.

Breck

#4

Feature request, ideas & comments are the reason why I put this up here. So thank you for the feedback. I threw thi

Model compilation errors are displayed in a red colored “flash message” similar to the “Model is compiling” box. There is a generic error displayed if the server experienced any hiccups. That was the case here. I was logging too much stuff and ran out of space. Its fixed now.

I did add a check for the input data. The only supported format currently is JSON, which is something I probably should list somewhere.

The caching feature that httpstan provides would shine here. I am guessing that there would be at least some matches out of the 40 models (or it would match the models from people that took the same class before them).

Your model probably compiled the last time, the server was just unable to respond. If you try compiling it again it should be instant. If anyone using this web app would compile the same model the compilations would be instant for them as well.

Not really, thank you. If there is interest I will definitely move this along faster. My current idea is to try and support most visualization that are provided by bayesplot.

#5

Rok,
Is there a repo where I can generate issues instead of making a mess here on discourse? Again thanks.

Breck

#6

Yes, I setup a repo on github: https://github.com/rok-cesnovar/cloudstan-experiment

Thanks.

#7

Hi–I tried Cloudstan and I have some issues / feature requests:

  1. Could there be an interface to R? That would be convenient. As it is, I needed to first work in R on my desktop, then convert the data to JSON format, then copy-and-paste into the Data input window, and then it didn’t work! (see info below)

  2. It seems to just run 1 chain, which will create problems for practical use. Can it be set to run 4 chains and then do convergence monitoring?

If this could all work, it would be huge, I think!

Stan program:

data {
int N;
vector[N] y;
vector[N] x;
}
parameters {
real a;
real b;
real<lower=0> sigma;
}
model {
y ~ normal(a + b*x, sigma);
}

–

JSON file:

{“x”:[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100],“y”:[9.1269,-47.4086,57.868,-40.9684,20.4094,3.9589,-36.5922,39.8938,-43.7822,10.5357,39.8121,55.5456,43.1374,51.8758,50.6403,120.7135,47.2974,144.8674,144.6857,69.5459,39.6452,103.3713,104.9762,46.3694,23.1451,101.9823,79.3229,102.4158,12.6294,112.5222,131.2641,82.7752,99.5206,49.2659,195.2491,158.1527,124.5499,147.5908,52.3644,213.7,38.4534,89.967,185.4385,146.365,157.424,171.4865,125.7648,145.9285,214.7259,164.4298,99.7023,61.9154,132.5432,174.5351,193.5842,191.5415,248.8224,124.6395,124.9634,155.6249,207.237,172.8277,149.2687,182.4149,181.9576,182.6331,196.8433,205.2649,292.506,252.3278,236.9424,202.5592,284.0804,248.8951,221.6069,139.9184,256.2312,309.3456,284.7533,198.8171,305.7594,200.5142,215.9679,217.356,180.8665,203.5907,246.2932,275.3513,221.6917,288.6411,271.8738,288.1322,208.0077,395.8834,223.6855,330.6136,339.3635,275.6744,256.7296,361.3982],“N”:[100]}

Error message:

Sampling started…ERROR:
Error calling services function: Exception: mismatch in number dimensions declared and found in context; processing stage=data initialization; variable name=N; dims declared=(); dims found=(1) (in 'unknown file name' at line 2)
undefined

#8

Yes. I am working on optionally supporting using Cmdstan as a backend, which would mean you could use all inputs that are supported by Cmdstan.

Same as above, by using Cmdstan that should be easily achievable. I am not sure if httpstan, that I use for the Stan backend right now, supports multiple chains.

The issue is that N should not be defined as an array, so just using writing "N": 100 instead of "N": [100] should work.

#9

Thanks! Mitzi is developing CmdStanR (or maybe I’m getting the name wrong; it’s the R interface to CmdStan) so maybe that will work here.

Regarding the JSON problem, I used the toJSON() function to convert my R list to that JSON file. I wonder if Stan’s method of reading JSON files needs to be fixed in some way to account for this problem.

P.S. Having an interface to R would help because then we could also make graphs etc.

#10

If when using the toJSON() or write_json() functions from the R package jsonlite to write a list to a JSON file, one should pass auto_unbox = TRUE to those functions. That way, component N from that list will be written to the JSON file as "N": 100 instead of "N": [100].

1 Like