We discussed this project at the Stan meeting, and Andrew and Thel are going to discuss it.
Thel writes:
Thinking of the project, I think it would be good to collect some other examples of developer communities that have worked well. I think the usual result of wikis etc is that you end up with a lot of abandoned or lost material and a lot of issues with moderation. I think at least initially we’re going to need a fair amount of handholding from developers to ensure the examples are actually decent.
There’s also the matter of how this overlaps with the example-models repo, the case studies page, and the examples in the manual. Having four places, plus discourse, plus the mailing list, feels out of control.
One thing I’ve thought about is whether we could build a better story around stackoverflow and encourage people to put questions there.
I’ve been bugging stan users about doing case studies, which will hopefully be nice to get online. Unfortunately what I find is this very common limitation: users spend a lot of time modeling missing or censored data — there’s no way around doing that but I think the code they end up writing is quite ugly and hard to understand. This is going to partly be solved by language tools like log_mix and partly by an idiomatic style of model building, which is something we might hope emerges from the community. But we’re probably in a place with the community where we need to curate aggressively and make sure the small fraction of actually readable examples end up on the front page.
Mike B writes:
Just let me note that there’s really no way to decouple all of these issues. As Thel notes there is substantial overlap with existing projects, and putting something similar together with either require some of our efforts or likely duplication of work. I know you don’t want to distract us, but any new projects will end up affecting us either now or later. This is another manifestation of what Bob keeps saying about maintenance — linearly more projects requires more than linearly more work because they’re not all independent of each other.
Anyways, I’m not trying to discourage a project like this, just
trying to communicate the realities.
Andrew writes:
I think we should be able to do this in a way that reduces rather than increases the labor of Bob and others here.
andrewgelman Developer
November 12
We discussed this project at the Stan meeting, and Andrew and Thel are going to discuss it.
Thel writes:
Thinking of the project, I think it would be good to collect some other examples of developer communities that have worked well. I think the usual result of wikis etc is that you end up with a lot of abandoned or lost material and a lot of issues with moderation. I think at least initially we’re going to need a fair amount of handholding from developers to ensure the examples are actually decent.
There’s also the matter of how this overlaps with the example-models repo, the case studies page, and the examples in the manual. Having four places, plus discourse, plus the mailing list, feels out of control.
One thing I’ve thought about is whether we could build a better story around stackoverflow and encourage people to put questions there.
I’ve been bugging stan users about doing case studies, which will hopefully be nice to get online. Unfortunately what I find is this very common limitation: users spend a lot of time modeling missing or censored data — there’s no way around doing that but I think the code they end up writing is quite ugly and hard to understand. This is going to partly be solved by language tools like log_mix and partly by an idiomatic style of model building, which is something we might hope emerges from the community. But we’re probably in a place with the community where we need to curate aggressively and make sure the small fraction of actually readable examples end up on the front page.
Mike B writes:
Just let me note that there’s really no way to decouple all of these issues. As Thel notes there is substantial overlap with existing projects, and putting something similar together with either require some of our efforts or likely duplication of work. I know you don’t want to distract us, but any new projects will end up affecting us either now or later. This is another manifestation of what Bob keeps saying about maintenance — linearly more projects requires more than linearly more work because they’re not all independent of each other.
Anyways, I’m not trying to discourage a project like this, just
trying to communicate the realities.
Andrew writes:
I think we should be able to do this in a way that reduces rather than increases the labor of Bob and others here.
The wiki getting out of date and the example models getting out
of date are both symptoms of the super-linear growth to which
Michael alludes. As we have more things (e.g., more example models,
in whatever form), then when we change things (like switching to =
from <-), then it’s more work to update them. That’s where the
quadratic growth comes in. Each new thing has to sweep over everything.
The only way to combat it is modularity and stability.
If we write a bunch of example model scripts and then go and change
RStan or switch the input format from the ad-hoc dump we use now
to JSON or something, then we have to take another pass which will
take time proportional to the number of examples we have.
Scikit-learn is a Python package for scientific computing which has
perhaps ~50 times more users than Stan (14,000 Github stars). Its
"Community" section consists of the following links:
Mailing List: for both users and developers (very low traffic!)
I’m shocked at how low traffic their mailing list is. One possible
explanation is that their documentation is so good that it takes off
some of the stress.
The biggest win is going to be modularizing the model code. Something like:
Maintain example models separately and automatically run them and generate output (figures and numerical output) that should allow at least for basic diagnostics if not regression testing (yet).
Write tests against that code when possible
Build user-facing documentation by incorporating that code/figures/etc…
I’m doing something like this with cmake on a smaller project but I don’t know if it’s the right tool. It’s great for the c/c++ parts but I end up having cmake run a lot of shell commands that are not really portable beyond Linux/OS X.
Just to clarify: It should not be the job of Bob or Thel or Stan Core to update models on the “Wiki” or “Facebook.” All models will come with dates which should indicate which version of Stan was being run. Future users of these models can feel free to update them (the “wiki” feature). Thus, if someone submits a model that nobody cares about, it may suffer from code rot but who cares. If someone submits a model that’s useful, the community will keep it up to date.
Indeed, one reason for doing this “Wiki” or “Facebook” thing is to take some of the burden of this model updating away from Stan Core.
If it could be set up so that contributed models could be auto-tested, that would be cool. This fits in with the idea that I had, of the submission process following a template.
A
Are you thinking one of the new hires will build this
thing then if not any of the people you mentioned?
They’ll need to build the site, find the compute cycles
to host it, then maintain it going forward. We might
be able to offload some testing cycles to an open-source
CI service like Travis.
Maintenance going forward involves administering the
hosting solution, updating any config or setup as services
change, and maybe reorganizing as we go and figure out what
we really want to do (say adding an organizing structure,
search, etc.)
The auto-testing would be something extra (“that would be cool,” as I wrote). For now I’m thinking of something hosted on Github or something like that, which does not run the Stan models, it just stores them, allows people to upvote make comments, etc. A Wiki or Facebook. So no big compute cycles needed.
A
I still don’t see how to host it and vote. Did
you mean literally starting a Facebook page
or literally using GitHub or is Facebook just
a proxy for liking and GitHub for commenting
and Wikis. I don’t see how the pieces would
fit together without any effort or hosting on
our side.
One of the Skikit learn people just moved to Columbia.
Stackoverflow can take a lot of load off of a mailing
list. We’d probably have to spread our list across Stackoverflow
and Crossvalidated.
That Gaussian mixture model is API doc. I agree that putting
ours in a better format and putting it online would be helpful.
Along with all the rest of our doc.
The bit of their doc that’s more relevant to what Andrew was
thinking about are their tutorials:
I don’t know how facebook works at all except that I’ve heard it has “likes.” Wikipedia is a model because it has user editing. Also I like the idea of comments. I think Facebook has that too.
What I want to do is talk with Thel about what I’m looking to do, and she can build it.
Or if Thel doesn’t want to build it, someone else can. Of course it will take effort to build. But any hosting would cost very little. I’m just talking about a place for people to post models, comment on them, edit them, and share them, along with various other things such as tagging. Later on we can talk about building something with auto-testing or whatever. I think what I’m talking about will be great for us and great for the user community and, once it’s set up, shouldn’t require too much maintenance. And there’s no reason it should interact in any difficulty ways with the Stan code. There are a lot of people fitting Stan models nowadays and it would be great for people to post these and share them with each other. Right now this happens occasionally on the stan-users list when people have questions. But what would be really great is if everyone’s successful models get posted too.
A
Wikipedia has a single target that multiple people can edit.
Someone is there managing the edit wars. Same as if we host
a page on Wikipedia and let people edit it.
Blogs have comments sections.
arXiv lets you post pdfs and then update them with versioning.
GitHub lets multiple people interact on code. And it has
issue trackers with comments. And it lets you comment on code
directly. But it’s not set up in a way that’d be easy for
Stan users to interact with and I don’t think it’s right for
this task.
Jupyter gives you a way to mix code and text, as does knitr.
These seem like obvious targets for rendering and making
reproducible code.
They’ll need to build the site, find the compute cycles
to host it, then maintain it going forward. We might
be able to offload some testing cycles to an open-source
CI service like Travis.
Having all the example models auto-building would require some cycles.
But perhaps there’s a company which would foot the bill? I suspect an
email to the right person at Rackspace or Google (Cloud) or Amazon or
even Digital Ocean (they’re in NYC, IIRC) might produce sufficient
virtual servers for Stan’s needs.
So just to doc the relatively simple framework we talked about on Tuesday:
An HTML upload form where a user uploads model_name.stan, and optionally model_name.data.R`` andmodel_name.init.R```, and fills in a few forms with title, authors, links to supplementary material, and descriptive comments.
Upon uploading the server creates a branch on some GitHub repository with the files and a formatted text file with all of the additional info, and then attempts a pull request.
The pull request hooks into Jenkins or some other CI service which tries to compile and run the model for, say, 10 seconds using the latest tagged release of CmdStan.
Upon success the pull request is merged, an email sent to the submitter, and a discourse topic created which includes the text information and the files for people to download and inspect.
Upon failure the branch is deleted and the user receives a failure message.
This would be relatively straightforward to implement, at least for testing, using the resources and knowledge that we have already (or at least almost have). We can dogwood it for a bit to test and then see if it’s providing the service we want.
I just learned that currently Kaggle Kernels have time limitation. I received “The kernel was killed for running longer than 1200 seconds”.
It’s also often disconnecting faster for no reason.
I guess these limitations are to limit server costs.
Hi, my only concern here is the requirement of setting up a data object. But if it’s really true that this can be done in one line of R (or Python), then I’m cool with that. In that case, I don’t see the need for there to be any R content here. User would upload Stan programs and data which would get checked. The submitter could also upload R or Python (or Julia or whatever) programs too, but these would not be checked or run by our server; they’d just be there to be useful to later users.
A