Has there been any work on enabling Stan for Spark? Azure notebooks / Databricks?
2 Likes
What is an Azure notebook/Databrick?
Microsoft Spark.
I think Stan would have a lot of application in transaction analysis, but sampling a space with multiple levels and millions of observations is a challenge.
Oh haha, I actually assumed you meant Apache Spark. Sounds like a real Microsoft soup of services. :)
can Stan’s map_rect
function be used in this case?
I haven’t tried it, but from reading the guide, it seems like I would have to map the data specifically (which could be a lot of work in dynamic scenarios) and I wouldn’t be able to pool effects between differently mapped groups (which is a bigger deal). So unless I misinterpret, it wouldn’t work. Also hoping that it would be simpler overall.