Hi there! (I was redirected from github here).
I’m running a set of phylogenetic and non-phylogenetic brms models on a Ubuntu 24.04.4 server with brms version 2.23.0, cmdstanr version 0.9.0 and R version 4.6.1, using cmdstanr as the backend. These models were run with 8 parallel chain across 8 CPU cores with 4 threads per chain. I haven’t had any issues with these until today when the chains started to terminate mid run in some models. The error messages usually read something like
Warning: Chain 6 finished unexpectedly!
Warning: 1 chain(s) finished unexpectedly!
The remaining chains had a mean execution time of 4759.8 seconds.
Warning: The returned fit object will only read in results of successful chains. Please use read_cmdstan_csv() to read the results of the failed chains separately.Use the $output(chain_id) method for more output of the failed chains.
grep: write error: No space left on device
Error in data.table::fread(cmd = fread_cmd, colClasses = "character", :
External command failed with exit code 2. This can happen when the disk is full in the temporary directory ('/tmp/RtmpKTdGGu'). See ?fread for the tmpdir argument.
Calls: <Anonymous> ... <Anonymous> -> read_cmdstan_csv -> read_csv_metadata
Execution halted
But my temporary dir isn’t full.
Filesystem Size Used Avail Use% Mounted on
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 63G 0 63G 0% /dev
devpts 0 0 0 - /dev/pts
tmpfs 13G 1.5M 13G 1% /run
/dev/vda2 30G 27G 2.1G 93% /
securityfs 0 0 0 - /sys/kernel/security
tmpfs 63G 8.0K 63G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
cgroup2 0 0 0 - /sys/fs/cgroup
pstore 0 0 0 - /sys/fs/pstore
bpf 0 0 0 - /sys/fs/bpf
systemd-1 - - - - /proc/sys/fs/binfmt_misc
hugetlbfs 0 0 0 - /dev/hugepages
mqueue 0 0 0 - /dev/mqueue
debugfs 0 0 0 - /sys/kernel/debug
tracefs 0 0 0 - /sys/kernel/tracing
fusectl 0 0 0 - /sys/fs/fuse/connections
configfs 0 0 0 - /sys/kernel/config
/dev/loop0 62M 62M 0 100% /snap/aws-cli/2312
/dev/loop1 62M 62M 0 100% /snap/aws-cli/2308
/dev/loop2 74M 74M 0 100% /snap/core22/2411
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
sunrpc 0 0 0 - /run/rpc_pipefs
/etc/auto.data 0 0 0 - /data/mounts
tmpfs 13G 96K 13G 1% /run/user/541
/dev/loop6 51M 51M 0 100% /snap/snapd/27591
tmpfs 13G 120K 13G 1% /run/user/6190 # my user account
tmpfs 13G 96K 13G 1% /run/user/514
/dev/vdb 2.0T 391G 1.6T 20% /data/mounts/scratch1
/dev/loop4 51M 51M 0 100% /snap/snapd/27710
tmpfs 13G 96K 13G 1% /run/user/6072
/dev/loop5 74M 74M 0 100% /snap/core22/2437
The model formulae are as follows:
data <- read.csv("https://github.com/anoban/phd/blob/main/data/chapter2/FRED/subsets/continuous_raw.csv")
data$taxa <- data$binominal
data$F00679 <- scale(log(data$F00679))[, 1]
data$F00727 <- scale(log(data$F00727))[, 1]
data$F00709 <- scale(log(data$F00709))[, 1]
tree <- ape::read.tree("https://github.com/anoban/phd/blob/main/data/chapter2/uphylomaker/FRED4_1301.tre")
if(!ape::is.binary(tree)) tree <- ape::multi2di(tree)
stopifnot(all(tree$tip.label %in% data$binominal))
corrmat <- ape::vcv.phylo(phy = tree, corr = TRUE)
M2 <- brms::brm(brms::brmsformula(mvbind(F00727, F00679, F00709) ~ 1 + (1|taxa)) + set_rescor(TRUE), data = data, chains = 8, cores = 8, threads = 4, iter = 5000, warmup = 2500, backend = "cmdstanr")
M4 <- brms::brm(brms::brmsformula(mvbind(F00727, F00679, F00709) ~ 1 + (1|q|taxa)) + set_rescor(TRUE), data = data, chains = 8, cores = 8, threads = 4, iter = 5000, warmup = 2500, backend = "cmdstanr")
M1 <- brms::brm(brms::brmsformula(mvbind(F00727, F00679, F00709) ~ 1 + (1|gr(binominal, cov = corrmat)) + (1|taxa)) + set_rescor(TRUE), data = data, data2 = list(corrmat = corrmat), chains = 8, cores = 8, threads = 4, iter = 5000, warmup = 2500, backend = "cmdstanr")
M3 <- brms::brm(brms::brmsformula(mvbind(F00727, F00679, F00709) ~ 1 + (1|p|gr(binominal, cov = corrmat)) + (1|q|taxa)) + set_rescor(TRUE), data = data, data2 = list(corrmat = corrmat), chains = 8, cores = 8, threads = 4, iter = 5000, warmup = 2500, backend = "cmdstanr")
The chain terminations aren’t very consistent. Sometimes the model finishes whithout any issues but mostly it ends up failing because of most chains terminating unexpectedly. Tried re-running the same models with rstan as the backend but that results in the same error. Any help would be appreciated. Thanks!