Jenkins Updates, Issues & Requests

Developers
41

I’m not sure if it’s the same error we’ve already seen, but the Jenkins upstream performance test failed (https://jenkins.mc-stan.org/blue/organizations/jenkins/Stan/detail/downstream_tests/1188/pipeline):

make: mkdir: Exec format error
ar -rs test/libstanc.a test/stan/lang/grammars/expression_grammar_inst.o test/stan/lang/grammars/program_grammar_inst.o test/stan/lang/grammars/functions_grammar_inst.o test/stan/lang/grammars/whitespace_grammar_inst.o test/stan/lang/grammars/term_grammar_inst.o test/stan/lang/grammars/block_var_decls_grammar_inst.o test/stan/lang/grammars/statement_2_grammar_inst.o test/stan/lang/grammars/expression07_grammar_inst.o test/stan/lang/grammars/local_var_decls_grammar_inst.o test/stan/lang/grammars/indexes_grammar_inst.o test/stan/lang/grammars/bare_type_grammar_inst.o test/stan/lang/grammars/statement_grammar_inst.o test/stan/lang/grammars/semantic_actions_def.o test/stan/lang/ast_def.o
ar: ar: creating archive lib/stan_math/lib/sundials_5.1.0/lib/libsundials_idas.a
creating archive lib/stan_math/lib/sundials_5.1.0/lib/libsundials_nvecserial.a
ar: creating archive lib/stan_math/lib/sundials_5.1.0/lib/libsundials_cvodes.a
ar: creating archive test/libstanc.a
/usr/local/opt/llvm@6/bin/clang++ -Werror  -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT       -I lib/stan_math/lib/tbb_2019_U8/include -O0 -I src -I . -I lib/stan_math/ -I lib/stan_math/lib/eigen_3.3.3 -I lib/stan_math/lib/boost_1.72.0 -I lib/stan_math/lib/sundials_5.1.0/include -I lib/stan_math/lib/gtest_1.8.1/include -I lib/stan_math/lib/gtest_1.8.1 -I lib/stan_math/lib/gtest_1.8.1/include -I lib/stan_math/lib/gtest_1.8.1 -I lib/stan_math/lib/gtest_1.8.1/include -I lib/stan_math/lib/gtest_1.8.1       -DBOOST_DISABLE_ASSERTS               -Wl,-L,"/Users/Shared/Jenkins/gelman-group-mac/workspace/Stan_downstream_tests/lib/stan_math/lib/tbb" -Wl,-rpath,"/Users/Shared/Jenkins/gelman-group-mac/workspace/Stan_downstream_tests/lib/stan_math/lib/tbb"  test/test-models/stanc.o         -Ltest -lstanc -o test/test-models/stanc
ld: warning: directory not found for option '-L/Users/Shared/Jenkins/gelman-group-mac/workspace/Stan_downstream_tests/lib/stan_math/lib/tbb'
test/test-models/stanc src/test/test-models/performance/logistic.stan --o=test/test-models/performance/logistic.hpp
make -j6 test/performance/logistic_test failed
42

I’ve just noticed that somebody must have restarted that and tests passed, only the pass status is not reflected in the issue. Is that doable or should I restart everything from scratch?

43

If you just restart the upstream test in Stan that does not propagate down.

44

Its the first I have seen of this. This error occured on Windows only. Lets hope its a one-off occurance. If not we should apply the same fix (building libs first).

45

Shame… Anyway, I’ve restarted all from scratch.

1 Like
46

Oh yes, very annoying. You can restart Upstream tests in Math, but that is it. Its still a few hours of testing.

47

make: mkdir: Exec format error
Strange because this can only happen if mkdir could not be executed as in something else had a lock on it. Thanks for the report, I’ll keep an eye on it.

1 Like
48

I’m not being very lucky with that PR! :) This really looks like the old error: https://jenkins.mc-stan.org/blue/organizations/jenkins/Math%20Pipeline/detail/PR-1598/6/pipeline

1 Like
49

A PR to fix the permission denied issue in math here

1 Like
50

I have just restarted the upstream tests (https://jenkins.mc-stan.org/blue/organizations/jenkins/Math%20Pipeline/detail/PR-1636/6/pipeline) but the corresponding issue is still marked as failed (Add row vector, array and int_array construction utilities by mcol · Pull Request #1636 · stan-dev/math · GitHub) rather than having the yellow circle (tests are still running).

Will it be fine when tests finish (if they pass) or do I need to make something happen behind the scenes? I have the feeling I tried that already in the past and ended up having to restart the whole pipeline.

51

If the tests pass it will turn green. Not sure why it doesnt turn yellow either.

1 Like
52

Out of memory error is now fixed by the intel fix. Will apply the fix to the windows ec2 machines and put them back in action.

1 Like
53

I don’t know if it’s just a temporary hiccup, but since there seem to be a queue on the Windows tests, perhaps it’s worth raising. In https://jenkins.mc-stan.org/blue/organizations/jenkins/Stan/detail/downstream_tests/1255/pipeline I got:

Cannot contact gelman-group-win-new: 
hudson.remoting.ChannelClosedException: Channel "unknown":
Remote call on JNLP4-connect connection from 193.2.76.138/193.2.76.138:49678 failed.
The channel is closing down or has closed down
54

@serban-nicusor Something is wrong as no PR has finished tests for several hours. There is at least a problem with the windows machines or in reporting back results.

55

Yeah, it seem that both instances are down. I restarted the one in our lab, hopefully that frees the backlog a bit. The Columbia Windows machine seems to properly be down https://jenkins.mc-stan.org/computer/gelman-group-win2/

56

Hey @mcol sorry for the slow update but I’ve been off for past two weeks.
Will now send an email to have the other machine booted up.