Negative binomial generalized additive models; improving computation time

  • Operating System: Ubuntu
  • brms Version: 2.7.0
  • Amazon AWS Community AMI (Server: RStudio-1.1.456_R-3.5.1_CUDA-9.0_cuDNN-7.2.1_ubuntu-16.04-LTS-64bit, AMI: ami-0a1ff58b80bb08791, Ready to run RStudio server for statistical computation (www.louisaslett.com). vCPUs: 16)

Hi Paul,
I’m attempting to run a series of negative binomial generalized additive mixed models using a web server to run them quicker. I couldn’t run them on my own machine because they were too computationally intensive. Despite using 16 cores on the fast server (4 on my own machine), it hasn’t improved the run time at all. So my main question is, is there anything in my code below that signals a problem I may not be picking up on?

My response variable is animal “detections”. There are 673 non-zero detections; the rest are zeros. My offset variable is “logdays” (days since animal was tagged) and my zero-generating process is “minoff” (minutes during a 15-min sampling interval that the unit was off due to malfunction, possibly leading to false zero detections). “PIT” is the unique identifier for each animal and is specified as a factor that corresponds to those 673 detection events (it has a 0 when no animal was detected). “mnth” is the grouping variable and I specify that the “stages” should vary by this category in the formula below.

str(d1st)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame':	38634 obs. of  6 variables:
 $ stages    : num  0.78 0.759 0.698 0.656 0.595 ...
 $ detections: num  0 0 0 0 0 0 0 0 0 0 ...
 $ mnth      : Factor w/ 12 levels "1","2","3","4",..: 6 6 6 6 6 6 6 6 6 6 ...
 $ logdays   : num  0 0 0 0 0 0 0 0 0 0 ...
 $ minoff    : num  0 0 0 0 0 0 0 0 0 0 ...
 $ PIT       : Factor w/ 62 levels "0","4","17","20",..: 1 1 1 1 1 1 1 1 1 1 ...

# A tibble: 38,634 x 6
   stages detections mnth  logdays minoff PIT  
    <dbl>      <dbl> <fct>   <dbl>  <dbl> <fct>
 1 0.780           0 6           0      0 0    
 2 0.759           0 6           0      0 0    
 3 0.698           0 6           0      0 0    
 4 0.656           0 6           0      0 0    
 5 0.595           0 6           0      0 0    
 6 0.512           0 6           0      0 0    
 7 0.409           0 6           0      0 0    
 8 0.306           0 6           0      0 0    
 9 0.182           0 6           0      0 0    
10 0.0588          0 6           0      0 0 

# tidybayes - create df list
d1st <- tidybayes::compose_data(d1st)

# brmsfit formula
bf1st <- bf(detections ~ logdays + minoff + PIT + s(stages, by=mnth) + # stages standardized (mean=0,sd=1)
              (1| mnth), # grouping variable
            zi ~ minoff) # zero-generating process

get_prior(bf1st, data=d1st)

priors <- c(set_prior("normal(0, 1)", class = "Intercept") ,
            set_prior("normal(0, 1)", class = "b") ,
            set_prior("cauchy(0, 1)", class = "sd"), # got this from rethinking
            set_prior("normal(0, 1)", class = "sds" ), # for smoothing term
            set_prior("normal(0, 1)", class = "b", dpar = "zi")) # distribution of zi

model1st <- brm(bf1st,
                data = d1st,
                family = zero_inflated_negbinomial(),
                chains = 2, cores = 16, # only two chains to start out
                iter = 1000, warmup = 500, # keep low to start out
                sample_prior = TRUE,
                prior = priors,
               control=list(adapt_delta=0.99, max_treedepth=15)) # in case divergent transitions

This is the message I get from the console after compiling the model:

Start sampling
SAMPLING FOR MODEL '08907781fb6515e1e8f6d5342ff7558a' NOW (CHAIN 1).

Gradient evaluation took 0.081834 seconds
1000 transitions using 10 leapfrog steps per transition would take 818.34 seconds.
Adjust your expectations accordingly!


Iteration:   1 / 1000 [  0%]  (Warmup)

SAMPLING FOR MODEL '08907781fb6515e1e8f6d5342ff7558a' NOW (CHAIN 2).

Gradient evaluation took 0.081025 seconds
1000 transitions using 10 leapfrog steps per transition would take 810.25 seconds.
Adjust your expectations accordingly!

Do you have any recommendations for this model approach, structure, prior specifications, etc.? Do you think it’s just too many zeroes in general for this type of analysis? I’m currently using the Amazon AWS Community AMI hosted by Louis Aslett (www.louisaslett.com) and it’s my first time trying this out so I’m not entirely sure what I’m doing there either.

I’m all ears!
Thank you for your time and the wonderful brms package,
Denise

Alright, I resolved my own issue for the time being; the code works on a subsample of zeros (e.g., 2000 rows total) so I will continue on searching for ways to speed things up.