How much disk do prob function tests take in stan-dev/math?

My Macbook Pro starts with 90GB of free disk space, and then
reports I run out of disk space (and reports 500MB free disk space)
after I run this:

stan-dev/math> /runTests.py -j4 test/prob

… fan blows, takes a long time, prints a lot …

clang++ -I . -isystem lib/eigen_3.2.9 -isystem lib/boost_1.60.0 -isystemlib/cvodes_2.8.2/include -Wall -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DNO_FPRINTF_OUTPUT -pipe -Wno-unknown-warning-option -Wno-unused-function -Wno-tautological-compare -Wno-c++11-long-long -Wsign-compare -Wno-unused-local-typedef -ftemplate-depth=256 -O0 lib/gtest_1.7.0/src/gtest_main.cc test/prob/student_t/student_t_cdf_log_00003_generated_ffv_test.o -DGTEST_USE_OWN_TR1_TUPLE -Wc++11-extensions -Wno-c++11-long-long -isystem lib/gtest_1.7.0/include -isystem lib/gtest_1.7.0 -o test/prob/student_t/student_t_cdf_log_00003_generated_ffv_test test/libgtest.a lib/cvodes_2.8.2/lib/libsundials_nvecserial.a lib/cvodes_2.8.2/lib/libsundials_cvodes.a
ld: not enough disk space for writing ‘test/prob/student_t/student_t_cdf_log_00003_generated_fd_test’ for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [test/prob/student_t/student_t_cdf_log_00003_generated_fd_test] Error 1
make: *** Waiting for unfinished jobs…
ld: not enough disk space for writing ‘test/prob/student_t/student_t_cdf_log_00003_generated_ffd_test’ for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [test/prob/student_t/student_t_cdf_log_00003_generated_ffd_test] Error 1
ld: not enough disk space for writing ‘test/prob/student_t/student_t_cdf_log_00003_generated_ffv_test’ for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [test/prob/student_t/student_t_cdf_log_00003_generated_ffv_test] Error 1
fatal error: error in backend: IO failure on output stream.
make: *** [test/prob/student_t/student_t_cdf_log_00006_generated_fv_test.o] Error 1

… keeps going with parallel jobs blowing out memory and then gives up …

  • Bob

I don’t know anymore. I don’t run locally.

OK, I’ll just push off to the server then. Given it was
where I had problems last time, I thought I’d check before
pull requesting.

I’m going to do a pull request for the stan-dev/math issue now.
I’ve eliminated the last traces of math.h and the top-level namespace.

When the stan-dev/math pull passes everything but upstream, I’ll
create the matching upstream branch. I already have it here but
want to wait to make a pull request.

What should I do for the lib/stan_math commit in the
stan-dev/stan branch? Should I use the new stan-dev/stan branch
or just leave it where develop is?

  • Bob

Yes. Just leave the branch at develop for the math library.

It may help to set N_TESTS=750 or so in make/local. In that case, it looks like a 3-argument distribution like the Cauchy consumes about 0.75 GB, while a 2-argument distribution is about 0.33 GB. I think even with Wiener, that would fit in under 90GB. However, it is still totally impractical to run routinely without Yeti.

<captain obvious>It would take longer but use less memory if you ran fewer simultaneous jobs.</captain obvious>

I’m running out of disk space, not memory.

It also spins up my machine to super-hot levels, so I’m reluctant
to run those tests. So I just have to wait and submit and see
back from the integration server.

Is there a way to run individual distribution tests? I can’t find
anything in email and there doesn’t seem to be a help option for
./runTests.py

  • Bob

I think that multiple jobs are also requiring multiple disk writes, especially if you have memory swapping issues going on, so I’m surmising that fewer jobs should use less disk space as well.

Hi Bob, I took a pass at the doc once a while ago:

On the jenkins box, it’s taking 122 GB now.

We really need to break this apart. It used to take a lot less disk space when it was just reverse mode (with a lot less time), but it blew up when we extended the testing to forward mode and 2nd order autodiff.

We should talk at some meeting about how we can manage this.