I’ve recently been running some models in brms that take a prohibitively long time to run (only getting a couple of % through their iterations overnight, for example). I think the reason is simply because these models are highly intensive in terms of memory requirements as e.g., they are ordinal models with repeated measures across participants, and many other multilevel components (random effects and intercepts) to enable poststratification. for thousands of respondents.
My current setup is a 16GB RAM computer and I can see this is getting maxed out when I run these models. In general, can I expect that using a cloud service such as AWS would increase the computational power sufficiently that these very long models would become more practically feasible? What kind of speed increases can I expect, or am I misunderstanding how cloud computing and MCMC sampling works, and a powerful cloud computer won’t actually help?
I’ve been using AWS before with good results. Adding enough RAM and enough virtual CPUs can speed things up, and then you can use threading also to “chop up” the chains on several CPUs, i.e., see: Running brms models with within-chain parallelization
It’s easy to set up an environment on AWS, but since then I’ve moved to a similar set up offered to employees at Swedish universities so whenever I run bigger jobs I run them in the cloud (it’s neither sane nor cool to run it on a laptop :)
Thanks @torkar . Nice that it should be simple to do - and yes I’ve been using within chain parallelization on my laptop, but have had to buy a massive and noisy cooling pad to prevent overheating! Does AWS assign a certain number of cores or how does that work? Is it like, X amount of cores with lots of RAM available in each or something else? Sorry that these are kind of noob questions but I haven’t used AWS before