Rstan, parallel, log_prob from inside function

Interfaces
1

Trying to parallelise an optimization routine using rstan, splitting the data up over separate workers. It works well, but I can’t get it to work as part of a function – any pointers? The following code breaks when it is wrapped in the test function, but works fine if the function definition part is removed.
edit: @bgoodri hope you don’t mind being tagged…

test <- function(){
  
  st='
  data{
  real myval;
  }
  model{
  target+=myval;
  }'
  
  sm <- stan_model(model_code=st)
  
  library(parallel)
  cl=makeCluster(2,type='PSOCK')
  
  standat=list(myval=1.2)
  
  smf<-c() #placeholder so we can assign to it from a function
  
  clusterExport(cl,c('smf','standat','sm'))
  
  clusterApply(cl,1:2,function(x) {
    library(rstan)
    standat$myval <<- x #CHANGE STANDATA FOR EACH PROCESS
    smf <<- sampling(object=sm,data=standat,iter=0,chains=0,init=0,check_data=FALSE, #regenerate stanfit object
      control=list(max_treedepth=0),save_warmup=FALSE,test_grad=FALSE)
  }
  )
  
  parLapply(cl,1:2,function(x) log_prob(smf,upars=1.2[c()],adjust_transform=TRUE, gradient=TRUE)) #get log prob for each process
  
}

test()
2

My understanding of advanced R is limited, but this seems very fishy and may break due to some environment magic. When I use clusterApply I let the the function return the value I care about, and clusterApply returns the list of the individual results e.g.:

smf <- clusterApply(cl,1:2,function(x) {
    library(rstan)
    standat$myval <<- x #CHANGE STANDATA FOR EACH PROCESS
    
    sampling(object=sm,data=standat,iter=0,chains=0,init=0,check_data=FALSE, #regenerate stanfit object
      control=list(max_treedepth=0),save_warmup=FALSE,test_grad=FALSE)
  }

Hope that helps.

3

that was the magic bit that lets me parallelise though, otherwise there is a lot more serialization, repeated splitting of data, repeated model generation. I don’t think that is the issue I think it’s some rcpp pointer issue that I don’t understand, but I may be wrong :)

4

I think it is more that the rstan package only looks for stanmodel objects in particular places, and that may not be suitable for some setups. I think it would work if you compiled the model once where the auto_write = TRUE argument is specified and then let the child processes find that on the disk when you do stan_model("file.stan").