Context: In early January 2020, @rok_cesnovar proposed a method for serializing Stan output using JSON . This post tries to continue the discussion. Any one of the following serialization formats could be used to improve serialization of Stan output: YAML, Protobuf, Avro, CBOR. (JSON is not an opti…

I’m a little unclear on scope: Is this meant to be a serialization standard for a single chain or for multiple chains? Is it meant to represent full input config to allow replication? Is it meant to include all adaptation information to allow restarting? Are the tools flexible enough in their sch…

I found this discussion useful as it talks about the serialization formats of Avro and protobuf (and Thrift, from [insert adjectives here] Facebook): https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html But it’s 7 years out of date. Also, is the main goal …

Was HDF5 considered? This is very well established in the scientific community and its been around forever.

Thanks for posting this! What in particular do you like about CBOR? Is there a mature’ish C++ package that can work with this format (or at least a stable one)? I think we’d also have to check if it’s available in R and Python (there is a protobuff R package) I would guess the adoption is even less…

Yes, there’s a popular, mature C++ library for JSON which also supports CBOR: https://github.com/nlohmann/json (17.2k github stars)

The scope is output: everything emitted to a logger or a writer when one calls a sampling function in stan::services. (In a call they have names logger, init_writer, sample_writer, and diagnostic_writer). [image] Bob_Carpenter: This seems to contradict the previous point. Could you clarify? T…

[image] wds15: Was HDF5 considered? This is very well established in the scientific community and its been around forever. I haven’t seen HDF5 used for (incremental) serialization like CSV/JSON/YAML/etc, only for storage.

[image] wds15: Was HDF5 considered? This is very well established in the scientific community and its been around forever. We discussed HDF5 but compared to something like CBOR it cuts you off from being able to write simple custom parsers for output because it’s a very complex format. The C…

[image] sakrejda: We discussed HDF5 but compared to something like CBOR it cuts you off from being able to write simple custom parsers for output because it’s a very complex format. The C api seems good but the corresponding C++ wrapper has long-standing problems (had last I checked) that indica…

I’m guessing that h5py goes through the C api so avoids the C++ api problems, it wasn’t anything against HDF5, just that we would be leaning on the C++ API a lot and if it doesn’t get updated reliably in response to bugs that’s something to consider. My post was a long time ago so things may be bet…

Notes on Stan Output Serialization Options (YAML, Protobuf, Avro, CBOR)

Developers

mike-lawrence August 2, 2020, 6:31pm 10

Possibly I don’t have sufficient expertise in this area to understand what you’re getting at, but I have been using h5py for years and hdf5r (successor to h5) for a decent amount of time too, and both seem to be pretty solid. Or were you referring to something else?

Topic		Replies	Views
JSON Output for STAN Developers	30	1472	October 3, 2023
Proof of concept: Binary output format for cmdstan Developers	74	2019	April 8, 2026
Status of the IO re-factor? Developers	16	1550	September 18, 2020
Schema for callback writers: one step towards retiring CSV Developers	99	6009	August 14, 2017
Speed difference between rstan and cmdstan for a simple model CmdStan rstan , techniques	25	4115	November 7, 2021

Notes on Stan Output Serialization Options (YAML, Protobuf, Avro, CBOR)

Related topics