It sounds as if we need some scripts which allow us to evaluate models for our various projects (warmup / any other tuning).
I would propose to use the R package
batchtools for this for these reasons:
- splits data generation and fitting into separate steps => you can evaluate, for example, different warmup strategies on exactly the same data generated replication. So per replication you get paired data for different warmup schemes.
- the execution engine is modular. So this can run on a single machine with/without multi-core support - or it can run on various clusters (SGE/LSF/slurm)
- relatively good debugging support
The only negative thing is that batchtool creates some overhead such that each job should represent substantial computational effort.
Anyone else has thoughts on this?