Jonah made me do it Report: CmdStan Binary Output Format Fork with code: GitHub - sims1253/cmdstan: CmdStan, the command line interface to Stan

[image] sakrejda: Doing column-wise row-chunks works well and that’s how arrow deals with that. It plays nicely with vector instructions, etc… I think we should do the simplest thing for now. This can be a place we optimize later. We could even have a little utility that transposes the data i…

@ssp3nc3r which Stan version you are using? A couple releases ago, the default number of digits saved to CSV was increased to reduce the probability of constraint errors in generated quantities. I’m curious whether you still have these problems with the latest release? Also instead of changing STAN_…

I’m using 2.37 and have the issue, and before the significant default digits increased from 6 (I think), I manually set them to 8 but it didn’t help. These vectors are up to >100K so I assume it becomes really hard to ensure an exact sum when all those lose digits.

@scholz are you planning on turning this into a design document? I’m happy to help (or write it, if you aren’t able or inclined)

I got kinda swept away by life and forgot, thanks for reminding me. I have the design proposal like 80% done. Should be able to push it over the weekend.

[image] WardBrian: The reason it still is not an option is that using Arrow well would require a redesign of the stan::io::writer interface to capture more information than it currently does about shapes and types of variables. Strictly speaking that’s not true, I’ve previously shared an impl…

I’m not sure how relevant this is to the cmdstan discussion, but if you are interested in some prior work, nutpie uses zarr to store large traces directly to disk in a binary format, so that arviz can read the results in right away without having to fit everything into memory. For in-memory represe…

I am very familiar with the previous discussions. It would certainly be possible to fit Arrow into the existing callbacks interfaces, but it would inherently be missing some desirable features or face unnecessary implementation difficulty to do so, because the writer interface as it is currently des…

The R interfaces parse those missing details from the column names and my arrow writer did the same. Arrow lets you construct a scheme once so it’s not even a performance issue (and faster than dumping things to text anyway). Those aren’t unusual challenges for any writer implementation in Stan.

Proof of concept: Binary output format for cmdstan

Developers

scholz March 9, 2026, 11:05pm 52

Add binary output format RFC by sims1253 · Pull Request #58 · stan-dev/design-docs · GitHub for those that follow here :)

Using AI Generated Code

Topic		Replies	Views
Alternative .csv reader Developers maintenance , rstan	53	4133	September 11, 2018
JSON Output for STAN Developers	30	1184	October 3, 2023
Speed difference between rstan and cmdstan for a simple model CmdStan rstan , techniques	25	3772	November 7, 2021
Notes on Stan Output Serialization Options (YAML, Protobuf, Avro, CBOR) Developers	13	3312	July 14, 2021
CmdStan dataset loading speed Developers cmdstan	21	1455	February 14, 2020

Proof of concept: Binary output format for cmdstan

Related topics