I had to restart it because my slurm job timed out. I’m hoping to use that to do some additional diagnostics on the priors and model.
In the meantime, @stevebronder suggested in another of my threads on this model (sorry for the multiple threads, but it helps to keep issues separate as I work through getting the model running) that I switch from cmdstanr to cmdstan. I’m running the cmdstanr version with additional warmups and a higher adapt_delta.
Using cmdstanr, I was getting some messages during warmup that I ignored based on this thread. I get the same warnings running on cmdstan, just more of them (~3000 with cmdstan vs. ~100 with cmdstanr) and it ultimately ends with the below messages. I’m using the same starting seed and setup, as far as I can tell.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: normal_lpdf(OpenCL): Random variable[-1076144195, 1846113984] = 0, but it must be not NaN! (in '../mtc_equity_7.stan', line 115, column 4 to column 33)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: offset_multiplier_constrain: multiplier[1, 1] is -nan, but must be positive finite! (in '../mtc_equity_7.stan', line 91, column 2 to column 56)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: offset_multiplier_constrain: multiplier[1, 1] is -nan, but must be positive finite! (in '../mtc_equity_7.stan', line 91, column 2 to column 56)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: offset_multiplier_constrain: multiplier[1, 1] is -nan, but must be positive finite! (in '../mtc_equity_7.stan', line 91, column 2 to column 56)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: offset_multiplier_constrain: multiplier[1, 1] is -nan, but must be positive finite! (in '../mtc_equity_7.stan', line 91, column 2 to column 56)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: copy (OpenCL)->Eigen: clWaitForEvents CL_EXEC_STATUS_ERROR_FOR_EVENTS_IN_WAIT_LIST: Unknown error -14 (in '../mtc_equity_7.stan', line 114, column 4 to column 33)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
terminate called without an active exception
terminate called recursively
[c2420:1401362] *** Process received signal ***
/var/spool/slurmd/job41230661/slurm_script: line 28: 1401362 Aborted (core dumped) ./mtc_equity_7 sample num_chains=4 data file=mtc_data.json output file=output.csv num_threads=36 random seed=76543
The noted code blocks are:
stan
row_vector<lower=0>[8] sd1; // group-level standard deviations
row_vector<lower=0>[8] sd2; // group-level standard deviations
matrix<multiplier = rep_matrix(sd1, N_1)>[N_1, 8] r_1; // ln 91
matrix<multiplier = rep_matrix(sd2, N_2)>[N_2, 8] r_2;
stan
target += student_t_lpdf(sd1 | 3, 0, 2.5);
target += student_t_lpdf(sd2 | 3, 0, 2.5);
for (k in 1:8) {
r_1[, k] ~ normal(0, sd1[k]); // ln 114
r_2[, k] ~ normal(0, sd2[k]);
}