Issues with Bayesian Hierarchical Reinforcement Learning Model Fitting

Modeling
,
4

I have already completed the construction of a basic Rescorla-Wagner (RW) model, and there were no issues such as divergent transitions in this model.

Okay well that’s encouraging! And although it doesn’t feel like it, remember that Stan’s diagnostics and warnings are a feature, not a bug. When you look at more complex learning and decision-making models from previous papers, keep in mind that in some cases the authors were using simpler / outdated model fitting software that actually did not compute the correct posterior, but didn’t raise any warnings about it.

I will also try modifying the prior distributions of my model.

Yes, to this end it might be easier to explore the priors in base R using something like this:

nSubjects <- 30 # change as appropriate

mu <- rnorm(n = 4, mean = rep(0, times = 4), sd = c(1, 1, 1, 5))
sigma <- rcauchy(n = 4, location = rep(0, times = 4), scale = c(1, 1, 1, 5))

alpha_pr <- rnorm(n = nSubjects, mean = 0, sd = 1)
gamma_pr <- rnorm(n = nSubjects, mean = 0, sd = 1)
c_pr <- rnorm(n = nSubjects, mean = 0, sd = 1)
tau_pr <- rnorm(n = nSubjects, mean = 0, sd = 1)

alpha <- pnorm(mu[1] + sigma[1] * alpha_pr)
gamma <- pnorm(mu[2] + sigma[2] * gamma_pr)
c <- pnorm(mu[3] + sigma[3] * c_pr)
tau <- pnorm(mu[4] + sigma[4] * tau_pr) * 20

So you really see step by step what parameter values are implied by your priors for a single iteration of MCMC sampling.

Currently, one more issue I’m facing is that each run takes a very long time to complete. Are there any methods to improve the efficiency of my model fitting?

Using more informative priors could also help speed up the model fitting. Otherwise I don’t see anything specific in your Stan code that could help, but perhaps others here on the forum have more tips?

Another thing you could consider is to use the pathfinder algorithm to obtain a preliminary model fit, and then use that as the initialisation for a final model fit using MCMC sampling, see e.g. this thread, or Aki Vehtari’s birthdays case study for a usage example. But I believe this is only available in CmdStan, so you’d have to use CmdStanR rather than RStan.