I’m, using cmdstanr/cmdstan 2.26.1 to sample a ridiculously large multivariate model and it does so fine with 6 parallel chains on my system with the default metric, but when I specify the metric as dense, my ram usage increases until cmdstan crashes (cmdstan might not be “crashing”; it does move on to trying the next chain) even with serial chains before the warmup could seemingly even start (at least I think; re-running with save_warmup=T now). So I’m thinking that possibly the dense metric is dramatically more RAM hungry for large models like this (or just stick with the default diag metric); would that be correct? If so, then the solution for me would be to go buy more RAM, but I thought I’d double-check in case this actually signals a bug I should be reporting instead.
The dense metric estimates an NxN covariance matrix. It won’t work well if N gets very big just cause the covariance estimate will be very bad. I think maybe with the default warmup, you’d probably only want to use the dense metric if N is less than like 200 or so (I think the longest warmup window will be like 500 or something – so that’s how many draws you’ll get to estimate the NxN covariance matrix).
So I guess I probably shouldn’t even bother buying more RAM; my N is about 30k :P Back to diag! (And diag wasn’t giving bad results, I was just curious about dense)
@bbbales2 These covariance matrices wouldn’t by any chance be amenable to a hierarchical low-rank approximation?
(Just asking out of curiosity)
I think that’s already in how cmdstan does the estimation; at least the docs refer to it being regularized toward a diagonal matrix.
There’s plenty of room for improvement in how we estimate our dense mass matrices.
I made an attempt here: [1905.11916] Selecting the Metric in Hamiltonian Monte Carlo
This is the code for that regularization: https://github.com/stan-dev/stan/blob/develop/src/stan/mcmc/covar_adaptation.hpp#L27 . (Edit: I just share the link to the code to show it’s not doing anything exotic).