Model makes R crash

Dear list,

This model is crashing my R session on an ubuntu machine when is about to start sampling R suddenly closes. Any thoughts of what’s going on? Is it an ubuntu thing? The code is below.
The data is in a list form and I can’t find a way to get it in a format allowed for upload. Here’s the link to google drive to get the data if someone wants to check if it works on their machine

https://drive.google.com/open?id=0ByhtWmffNWl-SFdzclh0WDNlMEk

data{
int<lower=1> N;
int<lower=1> N_year;
int<lower=1> N_zip;
int rate[N];
real no2[N];
real co[N];
real o3[N];
real pm10[N];
real pm25[N];
real no2_sq[N];
real co_sq[N];
real o3_sq[N];
real pm10_sq[N];
real pm25_sq[N];
real no2_co[N];
real no2_o3[N];
real no2_pm10[N];
real no2_pm25[N];
real co_o3[N];
real co_pm10[N];
real co_pm25[N];
real o3_pm10[N];
real o3_pm25[N];
real pm10_pm25[N];
int zip[N];
int year[N];
matrix[N_zip,N_zip] Dmat;
}


parameters{
real a;
real bno2;
real bco;
real bo3;
real bpm10;
real bpm25;
real bno2_sq;
real bco_sq;
real bo3_sq;
real bpm10_sq;
real bpm25_sq;
real bno2_co;
real bno2_o3;
real bno2_pm10;
real bno2_pm25;
real bco_o3;
real bco_pm10;
real bco_pm25;
real bo3_pm10;
real bo3_pm25;
real bpm10_pm25;
real<lower=0> theta;
vector[N_zip] a_zip;
real<lower=0> etasq;
real<lower=0> rhosq;
real<lower=0> sigma;
}


model{
matrix[N_zip,N_zip] SIGMA_Dmat;
vector[N] lambda;
rhosq ~ cauchy( 0 , 1 );
etasq ~ cauchy( 0 , 1 );
sigma ~ cauchy( 0 , 1 );
for ( i in 1:(N_zip-1) )
for ( j in (i+1):N_zip ) {
SIGMA_Dmat[i,j] = etasq*exp(-rhosq*pow(Dmat[i,j],2));
SIGMA_Dmat[j,i] = SIGMA_Dmat[i,j];
}
for ( k in 1:N_zip )
SIGMA_Dmat[k,k] = etasq + sigma;
a_zip ~ multi_normal( rep_vector(0,N_zip) , SIGMA_Dmat );
theta ~ exponential( 1 );
bno2 ~normal(0,1);
bco ~normal(0,1);
bo3 ~normal(0,1);
bpm10 ~normal(0,1);
bpm25 ~normal(0,1);
bno2_sq ~normal(0,1);
bco_sq ~normal(0,1);
bo3_sq ~normal(0,1);
bpm10_sq ~normal(0,1);
bpm25_sq ~normal(0,1);
bno2_co ~normal(0,1);
bno2_o3 ~normal(0,1);
bno2_pm10 ~normal(0,1);
bno2_pm25 ~normal(0,1);
bco_o3 ~normal(0,1);
bco_pm10 ~normal(0,1);
bco_pm25 ~normal(0,1);
bo3_pm10 ~normal(0,1);
bo3_pm25 ~normal(0,1);
bpm10_pm25 ~normal(0,1);
a ~ normal( 0 , 5 );

for ( i in 1:N ) {
lambda[i] = a + a_zip[zip[i]] + bno2 * no2[i] + bco * co[i] 
+ bo3 * o3[i] + bpm10 * pm10[i] + bpm25 * pm25[i] + bno2_sq * no2_sq[i]
+ bco_sq * co_sq[i] + bo3_sq * o3_sq[i] + bpm10_sq * pm10_sq[i] 
+ bpm25_sq * pm25_sq[i] + bno2_co * no2[i] * co[i] + bno2_o3 * no2[i] * o3[i] 
+ bno2_pm10 * no2[i] * pm10[i] + bno2_pm25 * no2[i] * pm25[i] 
+ bco_o3 * co[i] * o3[i] + bco_pm10 * co[i] * pm10[i] + bco_pm25 * co[i] * pm25[i]
+ bo3_pm10 * o3[i] * pm10[i] + bo3_pm25 * o3[i] * pm25[i] 
+ bpm10_pm25 * pm10[i] * pm25[i];
lambda[i] = exp(lambda[i]);
}
rate ~ neg_binomial_2(lambda , theta);
}

generated quantities{
matrix[N_zip,N_zip] SIGMA_Dmat;
vector[N] lambda;
vector[N] log_lik;
vector[N] y_pred;

for ( i in 1:(N_zip-1) ) 
for ( j in (i+1):N_zip ) {
SIGMA_Dmat[i,j] = etasq*exp(-rhosq*pow(Dmat[i,j],2));
SIGMA_Dmat[j,i] = SIGMA_Dmat[i,j];
}

for ( k in 1:N_zip )
SIGMA_Dmat[k,k] = etasq + sigma;

for ( i in 1:N ) {
lambda[i] = a + a_zip[zip[i]] + bno2 * no2[i] + bco * co[i] 
+ bo3 * o3[i] + bpm10 * pm10[i] + bpm25 * pm25[i] + bno2_sq * no2_sq[i]
+ bco_sq * co_sq[i] + bo3_sq * o3_sq[i] + bpm10_sq * pm10_sq[i] 
+ bpm25_sq * pm25_sq[i] + bno2_co * no2[i] * co[i] + bno2_o3 * no2[i] * o3[i] 
+ bno2_pm10 * no2[i] * pm10[i] + bno2_pm25 * no2[i] * pm25[i] 
+ bco_o3 * co[i] * o3[i] + bco_pm10 * co[i] * pm10[i] + bco_pm25 * co[i] * pm25[i]
+ bo3_pm10 * o3[i] * pm10[i] + bo3_pm25 * o3[i] * pm25[i] 
+ bpm10_pm25 * pm10[i] * pm25[i];
lambda[i] = exp(lambda[i]);

log_lik[i] = neg_binomial_2_lpmf( rate[i] | lambda[i] , theta );
y_pred[i] = neg_binomial_2_rng(lambda[i], theta);

}

}

You need rstan::stan_rdump(names(datalist), file = "my.data.R", envir = as.environment(datalist)). But it is probably just a compiler settings things that can be fixed if you tell us what the error message is.

I’ve had situations where R would just crash suddenly during sampling. I believe this is caused by overheating/flaky hardware. My hardware is more or less ok most of the time, but in heavy computing it seems like it has some issues. I installed thermald and set it up to keep the temp down lower than default and I think this has improved things for me. This web page was helpful in setting up thermald.

https://wiki.ubuntu.com/Kernel/PowerManagement/ThermalIssues

Here’s are the messages I get before R crashes. Thanks

In file included from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/config.hpp:39:0,
                 from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/math/tools/config.hpp:13,
                 from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/core/var.hpp:7,
                 from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/core/gevv_vvv_vari.hpp:5,
                 from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/core.hpp:12,
                 from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/mat.hpp:4,
                 from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math.hpp:4,
                 from /home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/model/model_header.hpp:4,
                 from file4ee857971845.cpp:8:
/home/tyatabe/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/config/compiler/gcc.hpp:186:0: warning: "BOOST_NO_CXX11_RVALUE_REFERENCES" redefined
 #  define BOOST_NO_CXX11_RVALUE_REFERENCES
 ^
<command-line>:0:0: note: this is the location of the previous definition
cc1plus: warning: unrecognized command line option ‘-Wno-ignored-attributes’

There should be some more to it, like “memory not mapped”. That is usually caused by the Rcpp package being compiled with a different compiler or different compiler settings than those use to compile the Stan program. If so, it can be fixed by reinstalling Rcpp.

I installed thermald. The setting up I was not able to do though. It allegedly starts running after installed. I ran the model again, and it crashed R.

yeah, if it crashes right away rather than after a few minutes… it’s probably not a thermal problem, more likely some kind of compiler issue. Nevertheless, thermald does help keep your temperatures down, so it’s useful to have, even without extra config its default settings are useful.

another thing I’ve seen happen is memory corruption in the OS cache (again related to running hot during a long sampling session), so if on a Linux system

echo 1 > /proc/sys/vm/drop_caches

fixes your problem, then you should consider testing and/or replacing memory or debugging hardware in general.

I tried that command, and it spits "Permission denied"
Also I ran free -m and it showed me this. I think I have plenty of available memory.

          total        used        free      shared  buff/cache   available

Mem: 7906 1015 6258 123 633 6476
Swap: 8115 2367 5748

I re-installed Rcpp and still crashes.

that command must be run as root

it’s not so much about free memory as whether the virtual memory system has a corrupted version of R in a memory cache.

Thanks. I tried but still spits Permission denied with this two different attempts:

sudo echo 1 > /proc/sys/vm/drop_caches

sudo -i echo 1 > /proc/sys/vm/drop_caches

Not so sure what’s going on

sudo sh -c "echo 1 > /proc/sys/vm/drop_caches"

should do it, the issue is that it’s your shell not the sudo command which is in charge of the redirection, you need to run a root shell.

Thanks, the command actually runs. rstan would still spit the same error message though.

I think you are running out of RAM, or at least I did.

Doing a centered GP in 1250 dimensions is asking for trouble. I can get it to start by non-centering

a_zip = cholesky_decompose(SIGMA_Dmat) * a_zip_raw;

in the model block but then you run into the error

[2] "  Exception: []: accessing element out of range. index 1231 out of range; expecting index to be between 1 and 1230; index position = 1a_zip  (in 'model1b603b898b5d_crash' at line 103)"   
error occurred during calling the sampler; sampling not done                                                                                                                                    

due to zip having a maximum of 1251.

Thanks, Ben. I fixed the values of zip to have a max of 1230, and then I tried the Cholesky decomposition (the code is below), but it still crashed when “started” sampling. This time I created the .r file, but still is too heavy, so here’s the link to my google drive.

T

https://drive.google.com/open?id=0ByhtWmffNWl-b2VBMnZJNE1uMk0

data{
int<lower=1> N;
int<lower=1> N_year;
int<lower=1> N_zip;
int rate[N];
real no2[N];
real co[N];
real o3[N];
real pm10[N];
real pm25[N];
real no2_sq[N];
real co_sq[N];
real o3_sq[N];
real pm10_sq[N];
real pm25_sq[N];
real no2_co[N];
real no2_o3[N];
real no2_pm10[N];
real no2_pm25[N];
real co_o3[N];
real co_pm10[N];
real co_pm25[N];
real o3_pm10[N];
real o3_pm25[N];
real pm10_pm25[N];
int zip[N];
int year[N];
matrix[N_zip,N_zip] Dmat;
}


parameters{
real a;
real bno2;
real bco;
real bo3;
real bpm10;
real bpm25;
real bno2_sq;
real bco_sq;
real bo3_sq;
real bpm10_sq;
real bpm25_sq;
real bno2_co;
real bno2_o3;
real bno2_pm10;
real bno2_pm25;
real bco_o3;
real bco_pm10;
real bco_pm25;
real bo3_pm10;
real bo3_pm25;
real bpm10_pm25;
real<lower=0> theta;
vector[N_zip] a_zip_raw;
real<lower=0> etasq;
real<lower=0> rhosq;
real<lower=0> sigma;
}

transformed parameters{
vector[N_zip] a_zip;
matrix[N_zip,N_zip] SIGMA_Dmat;

for ( i in 1:(N_zip-1) )
for ( j in (i+1):N_zip ) {
SIGMA_Dmat[i,j] = etasq*exp(-rhosq*pow(Dmat[i,j],2));
SIGMA_Dmat[j,i] = SIGMA_Dmat[i,j];
}

for ( k in 1:N_zip )
SIGMA_Dmat[k,k] = etasq + sigma;

a_zip = cholesky_decompose(SIGMA_Dmat) * a_zip_raw;

}

model{
vector[N] lambda;
rhosq ~ cauchy( 0 , 1 );
etasq ~ cauchy( 0 , 1 );
sigma ~ cauchy( 0 , 1 );
a_zip_raw ~ normal(0,1);
theta ~ exponential( 1 );
bno2 ~normal(0,1);
bco ~normal(0,1);
bo3 ~normal(0,1);
bpm10 ~normal(0,1);
bpm25 ~normal(0,1);
bno2_sq ~normal(0,1);
bco_sq ~normal(0,1);
bo3_sq ~normal(0,1);
bpm10_sq ~normal(0,1);
bpm25_sq ~normal(0,1);
bno2_co ~normal(0,1);
bno2_o3 ~normal(0,1);
bno2_pm10 ~normal(0,1);
bno2_pm25 ~normal(0,1);
bco_o3 ~normal(0,1);
bco_pm10 ~normal(0,1);
bco_pm25 ~normal(0,1);
bo3_pm10 ~normal(0,1);
bo3_pm25 ~normal(0,1);
bpm10_pm25 ~normal(0,1);
a ~ normal( 0 , 5 );

for ( i in 1:N ) {
lambda[i] = a + a_zip[zip[i]] + bno2 * no2[i] + bco * co[i] 
+ bo3 * o3[i] + bpm10 * pm10[i] + bpm25 * pm25[i] + bno2_sq * no2_sq[i]
+ bco_sq * co_sq[i] + bo3_sq * o3_sq[i] + bpm10_sq * pm10_sq[i] 
+ bpm25_sq * pm25_sq[i] + bno2_co * no2[i] * co[i] + bno2_o3 * no2[i] * o3[i] 
+ bno2_pm10 * no2[i] * pm10[i] + bno2_pm25 * no2[i] * pm25[i] 
+ bco_o3 * co[i] * o3[i] + bco_pm10 * co[i] * pm10[i] + bco_pm25 * co[i] * pm25[i]
+ bo3_pm10 * o3[i] * pm10[i] + bo3_pm25 * o3[i] * pm25[i] 
+ bpm10_pm25 * pm10[i] * pm25[i];
lambda[i] = exp(lambda[i]);
}
rate ~ neg_binomial_2(lambda , theta);
}

generated quantities{
matrix[N_zip,N_zip] SIGMA_Dmat_1;
vector[N] lambda;
vector[N] log_lik;
vector[N] y_pred;

for ( i in 1:(N_zip-1) ) 
for ( j in (i+1):N_zip ) {
SIGMA_Dmat_1[i,j] = etasq*exp(-rhosq*pow(Dmat[i,j],2));
SIGMA_Dmat_1[j,i] = SIGMA_Dmat_1[i,j];
}

for ( k in 1:N_zip )
SIGMA_Dmat_1[k,k] = etasq + sigma;

for ( i in 1:N ) {
lambda[i] = a + a_zip[zip[i]] + bno2 * no2[i] + bco * co[i] 
+ bo3 * o3[i] + bpm10 * pm10[i] + bpm25 * pm25[i] + bno2_sq * no2_sq[i]
+ bco_sq * co_sq[i] + bo3_sq * o3_sq[i] + bpm10_sq * pm10_sq[i] 
+ bpm25_sq * pm25_sq[i] + bno2_co * no2[i] * co[i] + bno2_o3 * no2[i] * o3[i] 
+ bno2_pm10 * no2[i] * pm10[i] + bno2_pm25 * no2[i] * pm25[i] 
+ bco_o3 * co[i] * o3[i] + bco_pm10 * co[i] * pm10[i] + bco_pm25 * co[i] * pm25[i]
+ bo3_pm10 * o3[i] * pm10[i] + bo3_pm25 * o3[i] * pm25[i] 
+ bpm10_pm25 * pm10[i] * pm25[i];
lambda[i] = exp(lambda[i]);

log_lik[i] = neg_binomial_2_lpmf( rate[i] | lambda[i] , theta );
y_pred[i] = neg_binomial_2_rng(lambda[i], theta);

}

}

It seems that you need more than 16GB of RAM. Also, that whole generated quantities block has to go. When RAM is tight, you have to be using the loo.function method instead of the loo.matrix method.

Thanks, Ben. Is there way to run this thing without buying a new computer/more ram?

T

You can always rent the necessary hardware on Amazon EC2, etc.

OK. I’ll see if I can do that.
Thanks for helping me out with this one!