Spark

#1

Has there been any work on enabling Stan for Spark? Azure notebooks / Databricks?

2 Likes
#2

What is an Azure notebook/Databrick?

#3

Microsoft Spark.

I think Stan would have a lot of application in transaction analysis, but sampling a space with multiple levels and millions of observations is a challenge.

#4

Oh haha, I actually assumed you meant Apache Spark. Sounds like a real Microsoft soup of services. :)

#5

can Stan’s map_rect function be used in this case?

#6

I haven’t tried it, but from reading the guide, it seems like I would have to map the data specifically (which could be a lot of work in dynamic scenarios) and I wouldn’t be able to pool effects between differently mapped groups (which is a bigger deal). So unless I misinterpret, it wouldn’t work. Also hoping that it would be simpler overall.