Sorry to stir what I know is a bit of a bees-nest by bumping this topic, especially given my limited expertise relative to everyone else that has chimed in, but a recent complaint regarding slow cmdstanr::read_stan_csv performance reminded me of this topic.
During StanCon I learned of @ahartikainen and colleagues’ work on ArViz, which uses NetCDF as its core file format, which in turn is really just HDF5 under the hood (as far as I understand).
Seeing as there is NetCDF expertise among Stan/Stan-adjacent devs, and possible motivation for said devs to help Stan itself move to NetCDF to make their job over in ArViz easier, and where the format debate doesn’t seem to have any clear winners otherwise, should we pull the trigger and ask the NetCDF crew to start working on a cmdstan branch replacing CSV with NetCDF?
Obviously they have the freedom to create said branch on their own regardless of an invite, but I think it would be more likely to actually happen if they were assured that their work would be a welcome solution.
Thoughts? Any major downsides to NetCDF specifically that make this an absolute non-starter?