This topic comes up periodically on the forum and I would like to propose using StanCon 2020 as an excuse to survey the Stan user community. While unlikely to be a representative sample, having a few hundred developers and users of the software at hand provides lots of opportunities to learn what the community does with the software and where they hope it would go in the future feature wise. The survey could just as easily go out to the broader Stan community via the the discourse forum but I liked the idea of using an in-person meeting as a motivator to do this.
What models do you build with Stan?
What sub-discipline does your work normally fall under?
What is the biggest time bottleneck in your normal Stan development workflow?
What diagnostic, modeling or performance features do you like in other packages that aren’t in Stan?
What do your colleagues use for their analyses?
There are likely better versions of these and other survey questions from the folks who understand experimental design far better than I do.
My suggestion is to have a tablet at registration for people to be surveyed, make it part of getting a t-shirt with easy opt out.
There was a survey post Helsinki (2018 v2) but I don’t recall where the results are.
A survey happened for Cambridge (2019) as well but we didn’t publicize results because I wanted detailed feedback about tutorials and it didn’t support easy generalization.
I am happy to share in greater detail personally. Used Google forms which work ok.
I definitely would like to see those questions if you are comfortable with sharing. Also am somewhat curious how the last attempt got stuck so maybe we collectively could avoid that happening again.
So just IMHO, I think qualifying survey participants or inviting them to participate based on some stratifying criteria may make sense since I’d guess researchers are more likely to attend the conference than undergrads or folks from industry.
I’d imagine it’s the same for different fields (perhaps it conflicts with another conference for example) or perhaps its more likely to attract the more local participants. So stratifying by field or by geography may also be helpful.
I guess I’m just saying perhaps use prior knowledge to sample better :-)
I definitely agree that recording experience level (with Stan and overall), geography, and field are import for understanding who is using the software. From post meeting survey results after Asiilomar, the 2018 meeting had a decent mix of students and career researchers. Someone must have sociological research on technical conference attendance but that is another matter.
My general hope is to get as many responses as possible and then try to post-stratify if possible, but I am still trying to work out the protocol. There is the tricky issue of preferential weighting between groups by whatever category. But before we get to that point I’d just like to have a draft of the survey questions and present it at some point on the Stan-Dev call for feedback.
Just IMHO again but if you actively decide who to survey you have the benefit of recording non respondents (ie the people that don’t want to take part) which can be a major confounder. I mean you may find that only a particular sort volunteer themselves.