I would like to submit a case study on probabilistic knowledge structures:
Knowledge structure theory is a field in psychometrics that seeks to
provide procedures for the effective diagnosis of the knowledge state of a
student in a certain domain (such as algebra, geometry, physics, or
statistics).
On the case-study page it just says “please contact us through the Stan
Forums”. Apologies if I’m missing anything. I’d be happy for any feedback.
Hi Florian –
This is indeed how to do it! The next step after you get some feedback would be opening a PR against the stan website to add it to the case study page there; here is an example of what that entails (basically just adding the built html page and then some text to link to it)
Unfortunately, it is difficult to preview the html file you’ve placed in the repository on Codeberg directly. Do you have a website you could host it on temporarily to make viewing easier?
Thanks, @Florian_Wickelmaier, and welcome to the Stan forums.
The writeup looks great.
Should there be arrows on the knowledge structure graph? If they’re implicit, would you mind saying so in the caption? I’m not convinced by this rigid knowledge structure that’s just assumed and the idea that if you know something, you’ll always get the answer right. Does this structure imply that it would be inconsistent with the model to observe someone getting question d correct and getting either a or b wrong?
I see then that the model breaks into “careless error” and “real error” and “lucky guess”. Some comparison to something like an IRT-1PL model that has a continuous difficulty parameter (as opposed to careless/careful) and IRT-3PL, which has a per-item guessing parameter.
If the graph is always subsets of the problems, then it makes sense to define the lattice as being closed under union (it’s not closed under intersection, though). I take it Q is equal to the full set of questions always? I’d be consistent and write either {} or \emptyset everywhere.
What was the motivation for pinning guess and error probabilities at max 0.5 in the BLIM? I would suggest just using <lower=0, upper=1> and letting the prior and data take the value where it should go to be consistent with the data. If you don’t have much data, the uncertainty is likely to extend beyond the 0.5 threshold. When the threshold’s in place, that mass has to go somewhere and it gets pushed down, introducing bias relative to the (0, 1)-constrained version of the model.
I wasn’t sure what “outer fringe” meant in defining `K^0$.
Figure 5 is really nice, but I’m not sure what “predicted” means here. Is there a distinction between the disks (solid circles) and circles (just border)—there’s nothing in the key for circles?
I think it’d help the reader to say a bit more about what LOO is doing so they can interpret the number dump. I’m pretty sure LOO lets you’d dump out the ELPDs and standard errors in a single output. I think that may be what you dump out at the end, but I don’t know what NA means. I think you wound up just below the 95% significance threshold on differences. This is not surprising because LOO isn’t very good at telling binary data apart. If you had more data, they’d probably be significantly different. But then I would expect the more general model to fit very well with more data, too.
What does “LOO CV is OK” mean? And what’s the NA column on the last report? How can
Estimate SE
BLIM: elpd_loo -5853.5 38.6
MNM1: elpd_loo -5861.2 38.4
produce
model elpd_diff se_diff p_worse diag_diff diag_elpd
BLIM 0.0 0.0 NA
MNM -7.6 3.7 0.98
I don’t see how the standard error on the difference can be only 3.7 when the standard errors on the ELPD estimates were 38.
I think you have a couple stray header columns (diag_diff, diag_elpd).
When you say “Again, parameter recovery is not perfect.” I think this is misleading. We don’t expect it to be perfect other than asymptotically as we get more data. If we didn’t get variation within the posterior intervals, we’d know the posterior intervals were too wide. I have the same question about circles and disks in this plot.
Any guesses as to how Lakshminarayan came up with the “knowledge graph” in figure 9? Was it empirically from test results or just a guess based on intuition? That is, if I have my own problem, where does this graph come from? With no guessing, it seems like it’s imposing a strict impossibility on some data responses.
I don’t think continuous responses are unguessable (the answer turns out to be 0 or 1 or 1/2 or sqrt(2) or pi too much of the time for it to be unguessable). I’d be inclined to fit the guessing model with a strong not going to be guessable prior.
I’m not sure what I should be comparing in figure 10. Are the shared parameters intended to be identical between models? It looks like it.
Are there frequentist packages to fit these models and if so, do you get similar results? Are there advantages to the Bayesian approach you want to include in the summary?
If whatever you’re using for layout can handle it, it’d be nice to hide most of the plotting code behind a combo-box that lets you select to show/not show the code. Same for the data munging code. I’d also hide the raw LOO computations and just present the final comparison after the first time (or even the first time as the details don’t matter so much for the point of this case study).
I’d also suggest just following the style guide in the index, which can be applied automatically. As is, there’s a lot of stylized spacing here that’s a bit distracting (like indenting priors and aligning things vertically leaving lots of space inside parens).
In writing math out, constants like function names are conventionally roman (e.g., \textrm{beta} rather than \textit{beta}.
Thanks for using reasonable amounts of precision in reporting.
Rather than a separate code supplement, I’d suggest just putting whatever you think people will need in an appendix. This isn’t to say you shouldn’t link to a GitHub or other repo, just that it’s nice if these things can be self contained. I hate coming to the end of a paper I’ve downloaded and realize I didn’t get the supplementary material.
Whenever there’s a fixed value, like here in the model block,
pi ~ dirichlet(rep_row_vector(1.0, nstate));
It’s better to move the constant up to transformed data so it doesn’t need to get reallocated each time (Stan doesn’t need the .0 like some languages—the C++ compiler figures out what each scalar type should be exactly).
transformed data {
row_vector[nstate] ones_nstate = rep_row_vector(1, nstate);
}
...
pi ~ dirichlet(ones_nstate);
Now having said that, even better is to just remove this sampling statement for pi because it’s a uniform distribution and hence a no-op as the simplex type automatically applies a uniform distribution over values matching its constraints (like all Stan types).
If you need more speed, it’d be faster to recode this:
(1 - R) * diag_pre_multiply(log( beta), K' ) +
R * diag_pre_multiply(log1m(beta), K' )
as
diag_pre_multiply((1 - R) * log(beta) + R * log1m(beta), K');
But whenever you see yourself needing a transpose, the better thing to do is just store K in transposed form so you don’t need the transpose as you go. This is the only place K is used, so just declare the transpose of K (you can call it K_t or K_tr if there would otherwise be confusion with standard naming).
I would suggest 0 as the lower bound for data size. Then you can set 0 size data to do posterior predictive checks.
I think the check eta_filter[i, j] > 0 ? 0 : 1 is easier to understand as !eta_filter[i, j] or eta_filter[i, j] == 0. You could get clever with some floating point operations to vectorize, but I don’t think the obfuscation is worth saving some vertical space.
There’s a download link, which was slow, but it worked.
Thanks Brian and Bob for your feedback!
A preview of the html version is here:
https://fwickel.codeberg.page/BayesianPKS/
I will work on Bob’s comments and report back.