Optional data and parameters

Hi,
Is there a way to optionally include both data and parameters in a stan model? I have a model that I would like to allow random effects to be optionally included at a certain level. So I would need a way to optionally supply e.g. the priors information for the covariance matrix for the random effects and their design matrix, and also optionally define the random effects parameters. I’ve tried a few approaches, but each has issues so far:

Option 1:
If I wrap everything that relates to the optional part (Data, parameters etc.) in if statements checking a boolean (that I’ve supplied as part of the data) that indicates whether or not the data and parameters will be present, the model gives error message

PARSER EXPECTED: <one of the following:
  a variable declaration, beginning with type,
      (int, real, vector, row_vector, matrix, unit_vector,
       simplex, ordered, positive_ordered,
       corr_matrix, cov_matrix,
       cholesky_corr, cholesky_cov
  or '}' to close variable declarations>

Which prints out at the line relating to the first occurence of the boolean in the data section:

data{
...
  if(boo_b3_l){
      matrix[M,rl] Z3_l;     
      vector[rl] b3_ones; 
    real<lower=0> tau_Al_sig;  
    real<lower=0> tau_Al_nu;   
    real<lower=1> Omega_Al_eta;     
  }
...
}

Option 2:
Alternatively, if I try to supply the data linking to these optional random effects, but set the number of these random effects rl to zero, e.g. define and supply the design matrix Z3_l as an M by 0 matrix, and only use the boolean when the effects are being sampled, I get error message:

Chain 1: Unrecoverable error evaluating the log probability at the initial value.
Chain 1: std::bad_alloc
[1] "Error in sampler$call_sampler(args_list[[i]]) : std::bad_alloc"
[1] "error occurred during calling the sampler; sampling not done"

Overall

  • is it possible to include optional data and optional parameters?
  • what is a reliable appropriate way to do so?

Note: I know this particular model can probably already be fitted using existing code, however this is a learning example for me to build up to coding a complex model that cannot be fitted using current code.

Full code:

functions{
}
data {
  //dimensions of data
  int<lower=1> M;      //number of observations
  int<lower=1> N;      //number of individuals
  int<lower=1> K;      //number of individuals
  int<lower=1> pl;     //number long fixed
  int<lower=1> ql;     //number long indrand
  int<lower=0> rl;     //number long studrand
  
  //id variables
  int<lower=1> idnum[M]; 
  int<lower=1> studnum[M]; 
  
  //longitudinal outcome information
  vector[M] y1;        
  
  //longitudinal fixed information
  matrix[M,pl] X_l;     
  real<lower=0> beta_l_sig; 
  real<lower=0> sigma_e_sig;     
  real<lower=0> sigma_e_nu;      
  
  //booleans
  int<lower = 0, upper = 1> boo_b3_l; 
  
  //longitudinal indrand information
  matrix[M,ql] Z2_l;    
  vector[ql] b2_ones; 
  real<lower=0> tau_Dl_sig;     
  real<lower=0> tau_Dl_nu;         
  real<lower=1> Omega_Dl_eta;     
 
  //longitudinal studrand information
  if(boo_b3_l){
      matrix[M,rl] Z3_l;   
    vector[rl] b3_ones;
    real<lower=0> tau_Al_sig;      
    real<lower=0> tau_Al_nu;     
    real<lower=1> Omega_Al_eta;   
  }
}
transformed data{
  vector[ql] mu_b2;
  if(boo_b3_l){
    vector[rl] mu_b3;
  }
  mu_b2 = rep_vector(0,ql); 
  if(boo_b3_l){
   mu_b3 = rep_vector(0,rl); 
  }
}
parameters{
  vector[pl] beta_l;  
  real<lower=0> sigma_e;      
  
  matrix[N,ql] b2;   
  corr_matrix[ql] Omega_Dl;     
  vector<lower=0>[ql] tau_Dl; 
  
  if(boo_b3_l){
    matrix[K,rl] b3;  
    corr_matrix[rl] Omega_Al;    
    vector<lower=0>[rl] tau_Al;  
  }
}
transformed parameters{
  vector[M] linpred_y;  
  matrix[ql,ql] Dl;     
  if(boo_b3_l){
    matrix[rl,rl] Al;    
  }
  
  linpred_y = X_l*beta_l + (Z2_l .* to_matrix(b2[idnum,]))*b2_ones;
  Dl = quad_form_diag(Omega_Dl, tau_Dl);
  if(boo_b3_l){
    linpred_y=linpred_y + (Z3_l .* to_matrix(b3[studnum,]))*b3_ones;
    Al = quad_form_diag(Omega_Al, tau_Al);
  }
}
model{
  //Priors
  tau_Dl ~ student_t(tau_Dl_nu,0,tau_Dl_sig);     
  Omega_Dl~ lkj_corr(Omega_Dl_eta);            
  if(boo_b3_l){
    tau_Al ~ student_t(tau_Al_nu,0,tau_Al_sig);   
    Omega_Al~ lkj_corr(Omega_Al_eta);             
  }
  beta_l  ~ normal(0,beta_l_sig);    
  sigma_e ~ student_t(sigma_e_nu,0,sigma_e_sig);     
  
  //Likelihood
  y1 ~ normal(linpred_y,sigma_e);  
  for(i in 1:N){
    b2[i,] ~ multi_normal(mu_b2, Dl);
  }
  if(boo_b3_l){
    for(i in 1:K){
      b3[i,] ~ multi_normal(mu_b3, Al);
    }
  }
}

Any comments / suggestions would be greatly appreciated - thanks in advance.

I’m not sure about data, but this is possible for parameters (I think rstanarm and such depend on it: Short blog on conditional parameters in Stan)

Yeah that’s right. It’s essential for packages with pre-compiled Stan programs.

I think having data with size 0 is ok but rstan may require passing in a length(0) R object of the appropriate type (I forget). Should be doable though. I think we probably do it for the data block in rstanarm somewhere.

It is doable - the linked blog has examples with both. I hope you’ll find it useful.

Good luck with modelling :-)

1 Like

Hi all,
Thanks for the advice - it feels like I am on the right track. I think I’m supplying data of length or dimension 0, and using boolean to select when parameters should be updated, but I must still be missing somthing, as I’m still getting the following error:

SAMPLING FOR MODEL 'longconMA' NOW (CHAIN 1).
Chain 1: Unrecoverable error evaluating the log probability at the initial value.
Chain 1: std::bad_alloc
[1] "Error in sampler$call_sampler(args_list[[i]]) : std::bad_alloc"
[1] "error occurred during calling the sampler; sampling not done"

The stan code I am using:

functions{
}
data {
  //dimensions of data
  int<lower=1> M;   
  int<lower=1> N;  
  int<lower=1> K;  
  int<lower=1> pl;  
  int<lower=1> ql;  
  int<lower=0> rl;   
  
  //id variables
  int<lower=1> idnum[M]; 
  int<lower=1> studnum[M]; 
  
  //longitudinal outcome information
  vector[M] y1;    
  
  //longitudinal fixed information
  matrix[M,pl] X_l;    
  real<lower=0> beta_l_sig; 
  real<lower=0> sigma_e_sig;    
  real<lower=0> sigma_e_nu;    
  
  //booleans
  int<lower = 0, upper = 1> boo_b3_l; 
  
  //longitudinal indrand information
  matrix[M,ql] Z2_l;    
  vector[ql] b2_ones; 
  real<lower=0> tau_Dl_sig; 
  real<lower=0> tau_Dl_nu;    
  real Omega_Dl_eta;   
 
  //longitudinal studrand information
    matrix[M,rl] Z3_l;   
    vector[rl] b3_ones; 
    real<lower=0> tau_Al_sig[boo_b3_l ? 1 : 0];    
    real<lower=0> tau_Al_nu[boo_b3_l ? 1 : 0];
    real Omega_Al_eta[boo_b3_l ? 1 : 0];   
}
transformed data{
  vector[ql] mu_b2;
  vector[rl] mu_b3;
  mu_b2 = rep_vector(0,ql);  
  if(boo_b3_l){
   mu_b3 = rep_vector(0,rl);  
  }
}
parameters{
  vector[pl] beta_l; 
  real<lower=0> sigma_e;      
  
  matrix[N,ql] b2;   
  corr_matrix[ql] Omega_Dl;    
  vector<lower=0>[ql] tau_Dl;   
  matrix[K,rl] b3; 
  corr_matrix[rl] Omega_Al;   
  vector<lower=0>[rl] tau_Al;   
}
transformed parameters{
  vector[M] linpred_y; 
  matrix[ql,ql] Dl;   
  matrix[rl,rl] Al;    
  
  linpred_y = X_l*beta_l + (Z2_l .* to_matrix(b2[idnum,]))*b2_ones;
  Dl = quad_form_diag(Omega_Dl, tau_Dl);
  if(boo_b3_l){
    linpred_y=linpred_y + (Z3_l .* to_matrix(b3[studnum,]))*b3_ones;
    Al = quad_form_diag(Omega_Al, tau_Al);
  }
}
model{
  //Priors
  tau_Dl ~ student_t(tau_Dl_nu,0,tau_Dl_sig); 
  Omega_Dl~ lkj_corr(Omega_Dl_eta);      
  if(boo_b3_l){
    tau_Al ~ student_t(tau_Al_nu[1],0,tau_Al_sig[1]);     
    Omega_Al~ lkj_corr(Omega_Al_eta[1]);            
  }
  beta_l  ~ normal(0,beta_l_sig);    
  sigma_e ~ student_t(sigma_e_nu,0,sigma_e_sig);    
  
  //Likelihood
  y1 ~ normal(linpred_y,sigma_e);  
  for(i in 1:N){
    b2[i,] ~ multi_normal(mu_b2, Dl);
  }
  if(boo_b3_l){
    for(i in 1:K){
      b3[i,] ~ multi_normal(mu_b3, Al);
    }
  }
}

and an idea of the data being fed in via r

data_for_stan3<-list(M=57260,
                     N=2000,
                     K=4,
                     pl=ncol(X_L),
                     ql=ncol(Z2_L),
                     rl=0,
                     idnum=longmerged$id,
                     studnum=as.numeric(as.character(longmerged$study)),
                     y1=as.numeric(longmerged$Y),
                     X_l=X_L,
                     beta_l_sig=100,
                     sigma_e_sig=1,
                     sigma_e_nu=3,
                     Z2_l=Z2_L,
                     b2_ones=rep(1,ncol(Z2_L)),
                     tau_Dl_sig=1,
                     tau_Dl_nu=3,
                     Omega_Dl_eta=3,

                     Z3_l=matrix(0,nrow=57260,ncol=0),
                     b3_ones=rep(1,0),
                     tau_Al_sig=rep(1,0),
                     tau_Al_nu=rep(1,0),
                     Omega_Al_eta=rep(1,0),

                     boo_b3_l=0
)

I have run the model with all parameters included successfully, using this size of data, but haven’t been able to get a version with optional paramaeters and data (as above) working. Would anyone be able to point me to an area of the code that maybe I have defined badly, or that would cause problems with sampling?

1 Like

I unfortunately cannot run your example, as I can’t reproduce your data. It might be a bug in Stan as there is history of zero-length structures causing problems: https://github.com/search?q=org%3Astan-dev+std%3A%3Abad_alloc&type=Issues

It would be very helpful if you could distill that into the smallest model that exhibits this problem.

Thanks!

smallstanexample.R (2.1 KB) long_uni_MA_edit2.stan (4.0 KB) longmerged.csv (893.8 KB) survmerged.csv (22.6 KB)

Hi,
I’ve made a bit of an example - this one still doesn’t work for me, and tries to include optional data and parameters. The R file (“smallstanexample.R”) loads the data, held in the two .csv files, and tries to run the model held in the .stan file.
Hopefully this will help to exhibit the problem, as requested?
Any help or pointers would be great (again - I’m doing this to learn how to write own programs in stan rather than use existing packages and code)
Many thanks

I should also point out, the attached data is simulated, and so rather “clean” straight line longitudinal data!

Zero-sized correlation matrices will segfault. This is a known bug. Cholesky factors don’t have the issue.

parameters {
  ...
  choolesky_factor_corr[rl] Omega_Al_c;
}
transformed parameters {
  matrix Al_c = diag_pre_multiply(tau_Al, Omega_al_c);
  ...
}
model {
  ...
  for(i in 1:K){
    b3[i,] ~ multi_normal_chol(mu_b3, Al_c);
  }
}
2 Likes