# CmdStan dataset loading speed

**URL:** <https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511>\
**Category:** Developers\
**Tags:** cmdstan\
**Created:** [July 3, 2019, 9:45am UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511 "2019-07-03T09:45:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tadej](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/tadej/32/5607_2.png) [@tadej](https://discourse.mc-stan.org/u/tadej)\
**Post date:** [July 3, 2019, 9:45am UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/1 "2019-07-03T09:45:13Z")

</div>

@Erik_Strumbelj noticed that CmdStan takes quite some time to load large datasets. After checking with a profiler I found out that the bottleneck are conversions from strings to doubles.

We could avoid these conversions if some binary file format was used. We could either specify our own (maybe 1 binary dump file per vector/matrix?) or use something like [hdf5](https://www.hdfgroup.org/).

What are your thoughts @Bob_Carpenter, @seantalts, @syclik ?

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [July 3, 2019, 2:16pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/2 "2019-07-03T14:16:06Z")

</div>

I suspect that parsing speed can be increased a lot. There is, for  
example, a massive difference between using an optimized json reader and  
a more generic one. I believe one observes this on multiple platforms.

As for binary formats, there has been some discussion about defining a  
protocol buffer schema for draws. We couldn’t reach agreement about what  
it should look like. PyStan 3 uses the following protobuf schema:

> <https://github.com/stan-dev/httpstan/blob/master/protos/callbacks_writer.proto>

---

<div class="post-metadata">

**Author:** ![stevebronder](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/stevebronder/32/17242_2.png) [@stevebronder](https://discourse.mc-stan.org/u/stevebronder)\
**Post date:** [July 3, 2019, 2:41pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/3 "2019-07-03T14:41:18Z")

</div>

I think @mitzimorris would be the most knowledgeable on how to tackle this. I think we also previously discussed protobuf (there’s also cap’n proto from the guy who made protobuf)

[https://capnproto.org/cxx.html](https://capnproto.org/cxx.html)

---

<div class="post-metadata">

**Author:** ![tadej](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/tadej/32/5607_2.png) [@tadej](https://discourse.mc-stan.org/u/tadej)\
**Post date:** [July 3, 2019, 2:43pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/4 "2019-07-03T14:43:25Z")

</div>

I doubt parsing speed can be significantly increased. Profiling identified vast majority of time is spent in conversions to doubles. These are done using boost::lexical\_cast that claims to be faster than sscanf, which I would otherwise default to. If we wanted we could offload these conversions to GPU, but it would not be simple.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 3, 2019, 3:01pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/5 "2019-07-03T15:01:36Z")

</div>

> [@tadej](#):
>
> or use something like [hdf5](https://www.hdfgroup.org/)

what is the install burden to the user for hdf5? I tried to install NetCDF on Mac OSX and it was ~~a nightmare~~ took some time. presumably users who need this feature can handle the install, correct?

---

<div class="post-metadata">

**Author:** ![tadej](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/tadej/32/5607_2.png) [@tadej](https://discourse.mc-stan.org/u/tadej)\
**Post date:** [July 3, 2019, 3:24pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/6 "2019-07-03T15:24:53Z")

</div>

I haven’t used it before but there seem to be binary packages for many systems. Anyway we can make it an optional feature, so users that need this may need to spend some time with installation, but everybody else can ignore it.

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [July 3, 2019, 3:34pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/7 "2019-07-03T15:34:10Z")

</div>

We already have a json option, right? Why not go with binary json (or msgpack)?

cc. @seantalts did you experiment with these?

edit. How long does it take to load data vs run a model that has huge data?

btw is lexical\_cast really fastest way to create a double?

---

<div class="post-metadata">

**Author:** ![seantalts](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/seantalts/32/32_2.png) [@seantalts](https://discourse.mc-stan.org/u/seantalts)\
**Post date:** [July 3, 2019, 4:30pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/8 "2019-07-03T16:30:34Z")

</div>

I just benchmarked to find out that it was slower than necessary and that using ujson would be nearly as fast in one case - see

[https://github.com/stan-dev/stan/issues/2776](https://github.com/stan-dev/stan/issues/2776)

And

[https://github.com/stan-dev/cmdstanpy/issues/38](https://github.com/stan-dev/cmdstanpy/issues/38)

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [July 3, 2019, 4:37pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/9 "2019-07-03T16:37:28Z")

</div>

> [@tadej](#):
>
> What are your thoughts

My immediate thought is, “Here we go again.” This has been a perennial topic of discussion with no resolution of which format or schema within a format to use. I believe protocol buffers are the current top runner for technology. We need something that supports efficient streaming input and output.

I think the only way this will be cracked is if someone takes a lead in a design based around a common standard.

> [@ariddell](#):
>
> optimized json reader and a more generic one

We wrote our own JSON reader based on callbacks for efficiency (it doesn’t build a parse tree as most of the existing packages do). But it just uses built-in string to primitive number conversions.

I measured the straight-up I/O cost of converting strings. It’s about a factor of 100 more expensive than binary I/O. It can be even more at full precision (about 20 ASCII characters per number with exponent if everything’s fractional) or less if you’re dealing with small integers. At full precision, ASCII isn’t very compact. We could add compression at probably very little cost or even a win in some places as it cuts down on raw disk I/O.

> [@tadej](#):
>
> Profiling identified vast majority of time is spent in conversions to doubles.

> [@seantalts](#):
>
> I just benchmarked to find out that it was slower than necessary and that using ujson would be nearly as fast in one case - see
> 
> [stan::io::json parser is 10x slower than python's ujson · Issue #2776 · stan-dev/stan · GitHub](https://github.com/stan-dev/stan/issues/2776)

Thanks for profiling. Are they using a custom or faster ascii to double converter? I also asked on the issue about some other things and don’t want to just duplicate that discussion here.

---

<div class="post-metadata">

**Author:** ![rok\_cesnovar](https://avatars.discourse-cdn.com/v4/letter/r/7bcc69/32.png) [@rok\_cesnovar](https://discourse.mc-stan.org/u/rok_cesnovar)\
**Post date:** [July 3, 2019, 7:27pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/10 "2019-07-03T19:27:41Z")

</div>

> [@ahartikainen](#):
>
> edit. How long does it take to load data vs run a model that has huge data?

I have notes on two GLMs (bernoulli and normal) with 1 million observation and k = 50. In this cases the input data.R file was ~1GB.

For the normal GLM on the CPU that ran for 270s, 40s of this time falling on I/O, almost all of it on reading the input data file.

15% is a notable amount of time I would say. If/When we manage to paralellize it with TBB with a 4x speedup, the I/O quickly becomes 40% of execution time.

The bernoulli GLM runs for 740s so the 40s of I/O is less of a problem, but its still 5%.

Normal GPU GLM on the Radeon VII runs for 55s, so 70% of execution time is spent on I/O.

The bernoulli GPU GLM runs for 95s, meaning that 40% of its execution time is again spent on I/O.

---

<div class="post-metadata">

**Author:** ![seantalts](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/seantalts/32/32_2.png) [@seantalts](https://discourse.mc-stan.org/u/seantalts)\
**Post date:** [July 4, 2019, 5:10pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/11 "2019-07-04T17:10:46Z")

</div>

> [@rok\_cesnovar](#):
>
> 15% is a notable amount of time I would say. If/When we manage to paralellize it with TBB with a 4x speedup, the I/O quickly becomes 40% of execution time.

Agreed, I think this is an issue. @ahartikainen convinced me to look into Python’s `ujson` library and it came out nearly as fast as msgpack and bson in some simple tests, so I think we could eat a ton of low-hanging fruit by just using some fast 3rd party JSON parsing library even if we don’t offer a binary format.

---

<div class="post-metadata">

**Author:** ![tadej](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/tadej/32/5607_2.png) [@tadej](https://discourse.mc-stan.org/u/tadej)\
**Post date:** [July 5, 2019, 10:24am UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/12 "2019-07-05T10:24:59Z")

</div>

I looked over suggestions here and in linked issues. In summary they are: ujson, BSON, MsgPack, Protocol Buffers, netcdf4, hdf5, Avro, Thrift, FlatBuffers, Cap’n Proto.

I think the need to specify schema and compile it into code for saving and loading files is unnecessary extra work in stan. That is a downside of Protocol Buffers, Thrift, FlatBuffers and Cap’n Proto.

I don’t think simply using another json parser can offer nearly as much speedup as a binary format so that is a minus for ujson.

Remaining options (BSON, MsgPack, netcdf4, hdf5 and Avro) all seem fine at first glance. Any preferneces? If there is a need I can write a benchmark to compare their performance.

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [July 5, 2019, 2:18pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/13 "2019-07-05T14:18:55Z")

</div>

I think CBOR should be on the list. It’s more of a standard than MsgPack  
or BSON. ujson is a library, not a format, right?

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [July 5, 2019, 6:10pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/14 "2019-07-05T18:10:56Z")

</div>

True. This could be tested against stan json reader

> **[esnme/ujson4c](https://github.com/esnme/ujson4c/)**
>
> Contribute to esnme/ujson4c development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![stevebronder](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/stevebronder/32/17242_2.png) [@stevebronder](https://discourse.mc-stan.org/u/stevebronder)\
**Post date:** [July 5, 2019, 10:28pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/15 "2019-07-05T22:28:43Z")

</div>

Adding on to json are these two

Seems easy to plugin

> **[nlohmann/json](https://github.com/nlohmann/json)**
>
> JSON for Modern C++. Contribute to nlohmann/json development by creating an account on GitHub.

Seems V fast

> **[lemire/simdjson](https://github.com/lemire/simdjson)**
>
> Parsing gigabytes of JSON per second . Contribute to lemire/simdjson development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [July 8, 2019, 6:51pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/16 "2019-07-08T18:51:01Z")

</div>

> [@tadej](#):
>
> I don’t think simply using another json parser can offer nearly as much speedup as a binary format so that is a minus for ujson.

Right. The bottleneck operations no matter what you do are:

1. ASCII to double and vice-versa,
2. file I/O
3. conversion to Python data structures

There’s no way to make the ASCII version of (1) competitive for full-precision floating point. I imagine a lot of speed differences in Python libs would come from (3).

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [July 8, 2019, 6:53pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/17 "2019-07-08T18:53:13Z")

</div>

> [@rok\_cesnovar](#):
>
> I have notes on two GLMs (bernoulli and normal) with 1 million observation and k = 50. In this cases the input data.R file was ~1GB.
> 
> For the normal GLM on the CPU that ran for 270s, 40s of this time falling on I/O, almost all of it on reading the input data file.

That’s in line with what I saw four or five years ago when I was evaluating how to best code linear regression in a Stan model, but I was running into cases where the I/O time dominated the fit time using optimization.

I don’t do enough I/O optimization to have a sense in how much of that 40s can be chopped off with a better format.

---

<div class="post-metadata">

**Author:** ![rok\_cesnovar](https://avatars.discourse-cdn.com/v4/letter/r/7bcc69/32.png) [@rok\_cesnovar](https://discourse.mc-stan.org/u/rok_cesnovar)\
**Post date:** [July 8, 2019, 8:25pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/18 "2019-07-08T20:25:16Z")

</div>

> [@Bob\_Carpenter](#):
>
> I don’t do enough I/O optimization to have a sense in how much of that 40s can be chopped off with a better format.

Neither do I.

My hunch is just that reading a GB of ascii data (or equivalent in any other format) should be faster than 40s on a modest computer.

---

<div class="post-metadata">

**Author:** ![tadej](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/tadej/32/5607_2.png) [@tadej](https://discourse.mc-stan.org/u/tadej)\
**Post date:** [July 18, 2019, 7:28am UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/19 "2019-07-18T07:28:53Z")

</div>

Many of those binary formats support saving a raw binary blob, but not something that is exactly a matrix or array of variables of same type. Is there any reason we would not want to use that for storing vectors and matrices? Since data is usually stored and loaded on the same machine (and big endian PCs are practically non-existant) endianness should not be a problem. We only have to specify that data must be stored column mayor. Or is there anything else I am missing?

---

<div class="post-metadata">

**Author:** ![fabio](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/fabio/32/935_2.png) [@fabio](https://discourse.mc-stan.org/u/fabio)\
**Post date:** [February 14, 2020, 2:01pm UTC](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511/20 "2020-02-14T14:01:21Z")

</div>

Maybe this would be’ useful?

> **[RcppSimdJson 0.0.1 now on CRAN!](https://www.r-bloggers.com/rcppsimdjson-0-0-1-now-on-cran/)**
>
> A fun weekend-morning project, namely wrapping the outstanding simdjson library by Daniel Lemire (with contributions by Geoff Langdale, John Keiser and many others) into something callable from R via a new package RcppSimdJson lead to a first tweet...

[Next page](https://discourse.mc-stan.org/t/cmdstan-dataset-loading-speed/9511.md?page=2)