Spark

EllenIAH · September 11, 2019, 2:06pm

Has there been any work on enabling Stan for Spark? Azure notebooks / Databricks?

increasechief · September 12, 2019, 1:21am

What is an Azure notebook/Databrick?

EllenIAH · September 12, 2019, 12:09pm

Microsoft Spark.

I think Stan would have a lot of application in transaction analysis, but sampling a space with multiple levels and millions of observations is a challenge.

increasechief · September 12, 2019, 1:24pm

Oh haha, I actually assumed you meant Apache Spark. Sounds like a real Microsoft soup of services. :)

mitzimorris · September 13, 2019, 12:08pm

can Stan’s map_rect function be used in this case?

EllenIAH · September 15, 2019, 9:31pm

I haven’t tried it, but from reading the guide, it seems like I would have to map the data specifically (which could be a lot of work in dynamic scenarios) and I wouldn’t be able to pool effects between differently mapped groups (which is a bigger deal). So unless I misinterpret, it wouldn’t work. Also hoping that it would be simpler overall.