Optimization of model to avoid 'Error: cannot allocate vector of size...'

Modeling
, ,
1

Hi there,

First off, sorry about yet another ‘Error: cannot allocate vector of size…’ thread. I have read through all the other threads and believe my rstan is working fine. The example(stan_model, package = "rstan", run.dontrun = TRUE) compiles and samples perfectly fine, as does the ‘schools’ example.

My problem may be my hardware and not the software itself. When I run my model (pasted below) I specify that I want to keep beta, lambda, sigma, log_lik, and sigma_total via the pars = c() argument and encounter the dreaded ‘Error: cannot allocate…’ message at the end of sampling. The model does work with sigma_total excluded from the pars argument. The model can also be run on low iterations, it just gets mad about Bulk and Tail ESS as expected (but is like 3.5Gb for 500 iterations). sigma_total for my dataset ends up being a matrix of about [450:450] for each iteration.

When I run the model as :

fit <- stan(file = './StanModel3.stan', data = stan_data6, warmup = 1000, iter = 5000, chains = 4, cores = 4, thin = 1,
             pars = c('beta', 'sigma', 'lambda', 'log_lik', 'sigma_total'))

I get the ‘Error: cannot allocate vector of size 39 Kb’ which seems to be a small amount of data to get mad about (unless this is for each iteration?)

When I thin the model just to see what happens (e.g. thin = 10), the error still appears at the end of sampling but is now 12 Kb and also added on:

Error in paste(new, collapse = "\n") : 
  could not allocate memory (0 Mb) in C function 'R_AllocStringBuffer'

My questions are:

  1. Is there a way to optimize the model somewhere that I am missing or collapse the sigma_total somehow in the generated quantities section? I want to keep the sigma_total parameter for later use in a LOO test.

  2. Is there a way to check if I am indeed hitting a hardware ceiling?

I am very new to modeling in stan and would love any feedback or help.

data {
  int <lower=0> N; // number of data points
  int <lower=0> K; // number of predictors 
  matrix[N,K] x; // predictor matrix
  vector[N] y; 
  matrix[N, N] d_mat; // sigma matrix
  matrix[N, N] A; // vcov matrix
      }

      parameters {
        vector[K] beta; // coefficients
        real<lower=0> sigma; // error
        real<lower=0,upper=1> lambda; // phylogenetic signal
      }

      transformed parameters {
        matrix[N, N] sigma_mat;
        matrix[N, N] sigma_total;

        sigma_mat = (1-lambda)*d_mat + lambda*A;
        sigma_total = sigma*sigma_mat;
      }

model {
  beta ~ student_t(3, 0, 10);
  lambda ~ uniform(0,1);
  sigma ~ student_t(3, 0, 2.5);
  
  y ~ multi_normal(x * beta, sigma_total);
}

generated quantities {
        real log_lik;
        log_lik = multi_normal_lpdf(y | x * beta, sigma_total);  
}