Jenkins Updates, Issues & Requests

Is there a problem with the gpu machine? I think this has been stuck for about half a day waiting for a gpu machine to free up:

Hey @mcol there were some custom performance tests running. I’ve let them run since yday since there wasn’t any load on the queue. I’ve now cancelled and the queue will execute all builds and PRs normally. Sorry for trouble.

I’d be happy to input new payment information into the account, and I know how to get that information from NumFOCUS, but I’ve been blocked for several months waiting on the new SGB to approve it. I didn’t realize this before but the old SGB only approved TWG Director funds for contractors, not for equipment or services (thanks @syclik).

I think these were broken, stuck infinitely compiling. I won’t have time to look into it.

@seantalts No worries, thanks! I’ve already opened a discussion with Steve and we’ll look into it.

1 Like

Not sure why I was tagged? The TWG Director was authorized to spend it without restriction on equipment or services.

Edit: see

The first $50k was allocated generally. The next $75k was allocated specifically for contractors. This was directly as a result of the request made to the SGB.

Yep, exactly - you pointed that out in an exchange a few months ago. I agree with your assessment, so I was thanking you for helping me follow protocol.

1 Like

The following build is stuck and I can’t get it to rerun (clicking the circular arrow button after logging in does nothing):

It’s fine now:


Thank you!

1 Like

It looks like authorization for the first $50K has lapsed and, as Sean points out, the $75K is restricted. We should resolve at the next @SGB meeting this coming Monday and arrange payment through NumFOCUS ASAP.


@serban-nicusor The issue with building sundials has appeared again (

ar: creating lib/stan_math/lib/sundials_4.1.0/lib/libsundials_kinsol.a
ar: unable to rename 'lib/stan_math/lib/sundials_4.1.0/lib/libsundials_kinsol.a'; reason: Permission denied
ar: creating lib/stan_math/lib/sundials_4.1.0/lib/libsundials_idas.a

Ah, thanks for posting @mcol.
@rok_cesnovar looks like the PR didn’t fix the issue :(

It did, we just didnt do the same fix on the Stan repository. I am on it.

1 Like

My bad, thanks for the quick reply and help!

1 Like

Docs can now be generated through a Jenkins job found here
It will build the docs, add redirects and create a PR in stan-dev/docs.
Also generates cmdstan manual which can be found in the job artifacts, example:

When done with all the docs the job will be there in more detail both usage and technical.

I’m not sure if it’s the same error we’ve already seen, but the Jenkins upstream performance test failed (

make: mkdir: Exec format error
ar -rs test/libstanc.a test/stan/lang/grammars/expression_grammar_inst.o test/stan/lang/grammars/program_grammar_inst.o test/stan/lang/grammars/functions_grammar_inst.o test/stan/lang/grammars/whitespace_grammar_inst.o test/stan/lang/grammars/term_grammar_inst.o test/stan/lang/grammars/block_var_decls_grammar_inst.o test/stan/lang/grammars/statement_2_grammar_inst.o test/stan/lang/grammars/expression07_grammar_inst.o test/stan/lang/grammars/local_var_decls_grammar_inst.o test/stan/lang/grammars/indexes_grammar_inst.o test/stan/lang/grammars/bare_type_grammar_inst.o test/stan/lang/grammars/statement_grammar_inst.o test/stan/lang/grammars/semantic_actions_def.o test/stan/lang/ast_def.o
ar: ar: creating archive lib/stan_math/lib/sundials_5.1.0/lib/libsundials_idas.a
creating archive lib/stan_math/lib/sundials_5.1.0/lib/libsundials_nvecserial.a
ar: creating archive lib/stan_math/lib/sundials_5.1.0/lib/libsundials_cvodes.a
ar: creating archive test/libstanc.a
/usr/local/opt/llvm@6/bin/clang++ -Werror  -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT       -I lib/stan_math/lib/tbb_2019_U8/include -O0 -I src -I . -I lib/stan_math/ -I lib/stan_math/lib/eigen_3.3.3 -I lib/stan_math/lib/boost_1.72.0 -I lib/stan_math/lib/sundials_5.1.0/include -I lib/stan_math/lib/gtest_1.8.1/include -I lib/stan_math/lib/gtest_1.8.1 -I lib/stan_math/lib/gtest_1.8.1/include -I lib/stan_math/lib/gtest_1.8.1 -I lib/stan_math/lib/gtest_1.8.1/include -I lib/stan_math/lib/gtest_1.8.1       -DBOOST_DISABLE_ASSERTS               -Wl,-L,"/Users/Shared/Jenkins/gelman-group-mac/workspace/Stan_downstream_tests/lib/stan_math/lib/tbb" -Wl,-rpath,"/Users/Shared/Jenkins/gelman-group-mac/workspace/Stan_downstream_tests/lib/stan_math/lib/tbb"  test/test-models/stanc.o         -Ltest -lstanc -o test/test-models/stanc
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/gelman-group-mac/workspace/Stan_downstream_tests/lib/stan_math/lib/tbb'
test/test-models/stanc src/test/test-models/performance/logistic.stan --o=test/test-models/performance/logistic.hpp
make -j6 test/performance/logistic_test failed

I’ve just noticed that somebody must have restarted that and tests passed, only the pass status is not reflected in the issue. Is that doable or should I restart everything from scratch?

If you just restart the upstream test in Stan that does not propagate down.

Its the first I have seen of this. This error occured on Windows only. Lets hope its a one-off occurance. If not we should apply the same fix (building libs first).

Shame… Anyway, I’ve restarted all from scratch.

1 Like