Issue with behrens volatility bayesian reinforcement learning model

Modeling
, , ,
1

Dear Stan expert, I am trying to use Behrens volatility model in Stan, and it is always failed in the Initialization steps. Can someone check my code? Thank you very much.
The math detail of volatility model can be found in here:https://static-content.springer.com/esm/art%3A10.1038%2Fnn1954/MediaObjects/41593_2007_BFnn1954_MOESM7_ESM.pdf
page 5 and 6 formula 1 and 2


data {
  int<lower=0> N; // trial number
  int y[N];
}

parameters {
  real k;
}

model {
  real r[N];   //reward probability
  real v[N];  //volatility
  k~cauchy(0,3);
  for(t in 1:N){
    if(t == 1){
      r[t] ~ uniform(0,1);
      v[t] ~ uniform(0,1);
    }
    else{
      v[t] ~ normal(v[t-1],exp(k));
      r[t] ~ beta_proportion(r[t-1],exp(k));
    }
    y[t] ~ bernoulli_logit(r[t]);
  }
  
}

And if I fit the model with some simulate data, there always some warning

o/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Rejecting initial value:
Chain 2   Error evaluating the log probability at the initial value.
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Rejecting initial value:
Chain 2   Error evaluating the log probability at the initial value.
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Rejecting initial value:
Chain 2   Error evaluating the log probability at the initial value.
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Rejecting initial value:
Chain 2   Error evaluating the log probability at the initial value.
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Rejecting initial value:
Chain 2   Error evaluating the log probability at the initial value.
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Rejecting initial value:
Chain 2   Error evaluating the log probability at the initial value.
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Rejecting initial value:
Chain 2   Error evaluating the log probability at the initial value.
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Exception: normal_lpdf: Random variable is nan, but must be not nan! (in 'C:/Users/lenovo/AppData/Local/Temp/RtmpWqbkxV/model-8a54e281649.stan', line 27, column 6 to column 35)
Chain 2 Initialization between (-2, 2) failed after 100 attempts. 
Chain 2  Try specifying initial values, reducing ranges of constrained values, or reparameterizing the model.
Chain 2 Initialization failed.
Warning: Chain 2 finished unexpectedly!

Warning: Use read_cmdstan_csv() to read the results of the failed chains.
 Fitting failed. Unable to print.
= Warning messages:
1: All chains finished unexpectedly! Use the $output(chain_id) method for more information.
 
2: No chains finished successfully. Unable to retrieve the fit.
1 Like
2

The problem is most likely that you declare r and v but never assign a value to them. Everything except parameters (declared in the parameters block) needs to be explicitly assigned a value before use (otherwise it is nan).

The sampling statement (y ~ distribution(params)) can seem to imply that you sample the parameter, but it is in fact just a shorthand for target += distribution_lupdf(y | params) - i.e. for adding the respective density to the posterior.

Maybe you wanted to have r and v as parameters? Additionally, if r and v are uniformly distributed between 0 and 1, you should declare them with appropriate bound (i.e. real<lower=0, upper=1> r[N]).

Best of luck with your model!

2 Likes
3

Thank you for responding me.

4

I have tried to implement this model for years.

And I have written a PyMC3 model which is too slow and a Turing.jl model that is hard to convergence.

Thank you for your code. I can implement an efficient one.

1 Like