My concern here is the false positive aspect. We can talk ourselves into tau being bad because we already know it should be bad, but if you looked at the plot without any prior bias would you think the other parameters look okay? Why should be be considered okay here? It seems like the non-divergence iterations bifurcate whereas the divergent ones don’t, so it’s not like they’re consistent or anything.
Also let me emphasize again that in practice it’s usually the correlation plots that help you identify what part of the model is causing trouble so I’m not sure how useful this plot will be. Ultimately, the test will be not applying to the 8 schools model but rather on new problems and see if it ends up being helpful.
Yes, one could do that. The only problem is that some values are closer to log-scale. So it would look like almost all the points are in one place.
Me too. Is it an artifact from the transformation or a real phenomenon?
I agree with this. In my opinion, this plot should be (if used) the first step to identify problematic areas.
This plot should be tested a bit more to see if it is really useful or not.
Also, do notice that the normalization function assumes that there are no duplicate points in the sample. If there are, they will be a shifted due to cumulative function. Not really a problem if one only does visual stuff (n>1000), but it should not be used anywhere else.
The file is ‘westbrook.R’. The data gets loaded up and model executed in the first 16 lines. The rest is diagnostics and plots. As the model is configured, you should get 50~ divergences (4 chains, 2000 iters each).
It’s an approximate GP technique I was testing out. The story is:
I took Russell Westbrook’s made-missed shots from the 2015 season (or at least most of it – not sure if the dataset is complete). It’s an r/nba think to talk about how in 4th quarters of games Westbrook turns into Worstbrook (as opposed to Bestbrook), so the idea is maybe we could estimate Westbrook’s shooting percentage not as a single number, but as a function of the game time (and see the difference in Bestbrook and Worstbrook)
It took me awhile to work out where the divergences were coming from because I wasn’t sure if it was the approximation that was causing the divergences, or something weird with the data (and an exact model would have caused divergences), or something else. Anyway, seems like a good level 2 crackme. (edit: and I sure would have appreciated something more than pairplots when I was debugging this)
Re the prominent sawtooth pattern. Each of the theta is probably a mixture of gaussians through the mu mean and tau sd. Let mu and tau be fixed. Then as a mechanical analog, thetas are independent harmonic oscillators with equal periods and random phases. In essence, balls on springs hanging from a stiff metal plate, bouncing up and down.
Now, un-fix mu by instead of dangling the plate from the ceiling on a rope, dangle it from a spring. Then the harmonic oscillators are all coupled together, their motion affects the motion of mu. When the oscillators all pull downward, the plate moves downward, when they all push upward, the plate moves upward. The coupling through mu causes partial synchronization of the oscillators. In order to have a constant mu, they need to have about half of them going up, and half of them going down. If mu has smaller posterior sd than thetas which it should (about 1/sqrt(8)) then for a diagonal mass matrix, it has 8 times the mass and its velocity which is p/m is 1/8 as much so it’s kind of constant relative to the oscillations of the theta.
This is all a guess. But I think it makes reasonable sense. I now think we need a Stan T-shirt with 8 balls dangling from a plate dangling from the ceiling.
Tau here corresponds to the stiffness of the springs. When tau is small, it means you don’t stretch the springs much compared to equilibrium position, so your oscillation amplitude is small. When tau is large then the springs are soft, and you can wiggle around a lot. So, imagine you’re going along with a tons of balls wiggling on springs, and then suddenly all the springs freeze up into hard stiff steel rods (tau goes to 0). BANG all the balls need to snap back to the center forcefully. If you’re simulating this, you need to accelerate the balls dramatically, and then move the balls with a LOT of velocity. For any reasonable step-size you will be unable to get your balls back to the right place, they will overshoot, which corresponds to them stretching a steel rod by a lot more than they would in reality, which corresponds to then applying a restoring force on the balls that is way way way too large, which corresponds to turning around and overshooting again, compressing the rod by way too much, which corresponds to turning around… and diverging out to infinity after a few cycles.
Finally, imagine that in addition to the spring forces on the balls, there are winds blowing around, so that the ball trajectories are randomly perturbed by additional forces. These additional forces are the forces from SPSA approximate gradients in my discussion here:
Now imagine each ball internally has some kind of computer mechanism and some mass it can wiggle around inside the ball using battery energy. We let it wiggle with random normal perturbations on regular, extremely fast intervals. This is the random noise I added to the momentum step in my sampler. Finally, we have all the ball computers wired together so they can detect their total energy. when their total energy drifts a little too low, they run their internal mechanism adding some velocity to all the balls in the direction of current travel, proportional to the current velocity. when the total energy is too high, they all apply a viscous drag on their spring to slow themselves down. This is the control viscous force.
Next, we take snapshots of the system at regular time intervals, but we throw away all the snapshots where the system didn’t have within epsilon of the right total energy that it had at the beginning of the run… Finally, at the end of all of it we randomly choose one of the remaining snapshots with probability proportional to exp(-TotalPotentialEnergyInSprings)…
That’s a mechanical description of my momentum diffusion approximate HMC I used in linked thread above.
In the upper plot, when we get a divergence, tau is near zero, and the thetas are all basically constant (horizontal green lines). In the ball and spring scenario this is due to the fact that tau is the RMS displacement of the balls from the average ball location. If tau drops to zero, all the springs harden into steel rods of the same length, and all the balls have to snap to the same place… and they do, but as I said the forces are enormous and we need a very small time-step to track these dynamics, but Stan takes constant time steps, so the simulation diverges.
The solution here is to tell the model that the springs are never steel rods of the same length, basically a zero avoiding prior on tau, which is well known in the 8 schools example. But I think the mechanical analogy is apt and can help debug more difficult models. I too think bayesplot should produce these parallel coord plots. They could be very useful. Note though that both a natural scale plot (or a log of a natural scale plot) and a probability scale plot could be useful (ie. both the upper and lower). The lower plot doesn’t preserve the flatness of the thetas during divergences which are essential to the story.
I don’t have the slightest idea what the approx_L function is doing, but I see that in the end it produces something where we do
inv_logit(approx_L * z)
I also see approx_L has some kind of lagged structure Ht[n] is a function of Ht[n-1].
and approx_L is linear in sigma. so I assume as sigma gets big z0 and z1 have to go to zero or we do something stupid like predicting that the player has binomial(1) or binomial(0) shot accuracy for essentially all time, and the data bounces us back away from there pretty hard.
In the end I decided it was some interaction of z and sigma that was causing the problem. You get these divergences when you fit GPs to data that looks flat (like if it’s just iid noise around the x-axis which is kinda like lengthscale going to zero). In the end I just fixed sigma to be a constant.
How’d you know to only look at the first 12 parameters? How’d you decide to look at z?
seems to do a decent job on the gp and 8 schools examples, when using the relative location of divergences as a guide… not sure whether I’m just getting lucky / overfitting though, as I expected the sd (function returns both) to be more useful :)
Thanks for this Charles, I just tried it for finding divergences in one of my models. I haven’t tried the parellel coordinates plot yet, maybe it it will point out the same thing. This identifies a funnel between two of my params as the lowest sd.
I added a feature to the function to specify (either by name or by numeric index) which params you want to look at. This was modified from the original version posted above but would be trivial to integrate into the new version. I also added a composite score ranking (rank the sd and loc and multiply them together.)
No problem. I’m actually thinking of writing up a blog post on debugging my model, would you mind if I reference my modified version as a gist on github? I’ll attribute in the blog post, and if you like I can put you in the author tag in the roxygen block.
Sure, sounds good. If you’re looking into large, slow, troublesome models, the stanWplot (stan with online trace plots) function I wrote (can find it as part of the ctsem package on github) might also be helpful.