Probability test broken locally in unspecified way

Developers
1

Hey,

I’m running the probability distribution tests (the generated ones) with the new operands_and_partials and one of them is failing and just exiting like so:

make -j8 test/prob/normal/normal_00000_generated_fv_test
make: `test/prob/normal/normal_00000_generated_fv_test' is up to date.
------------------------------------------------------------
test/prob/normal/normal_00000_generated_fv_test --gtest_output="xml:test/prob/normal/normal_00000_generated_fv_test.xml"
Running main() from gtest_main.cc
[==========] Running 800 tests from 100 test cases.
[----------] Global test environment set-up.
[----------] 8 tests from AgradDistributionNormal_fv_0/AgradDistributionTestFixture/0, where TypeParam = boost::mpl::vector<AgradDistributionNormal, boost::mpl::vector<double, double, stan::math::fvar<stan::math::var>, empty, empty, empty, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>
[ RUN      ] AgradDistributionNormal_fv_0/AgradDistributionTestFixture/0.CallAllVersions
test/prob/normal/normal_00000_generated_fv_test --gtest_output="xml:test/prob/normal/normal_00000_generated_fv_test.xml" failed
exit now (04/19/17 18:23:27 EDT)

Anyone know how to get more information? I assume I can go dig into INSTANTIATE_TYPED_TEST_CASE_P and start adding printlines but I am guessing this is common enough that some people might have some tips?

Thanks,
Sean

2

what branch are you on? I don’t have a problem with ops_partials.

3

ops_partials https://github.com/stan-dev/math/tree/ops_partials

4

So @syclik took the time to show me a few ways to debug tests, which I will detail here for posterity. These are pretty general but it’s nice to see how to specifically do these techniques with gtest and our codebase.

First off, it’s always good to be able to isolate the failure in as small of a test case as is possible. Dan tracked down the issue in one of the generated tests (test/prob/normal/normal_00000_generated_fv_test.cpp) by grepping for one of the strings in there, AgradDistributionNormal, leading us to test/prob/normal/normal_test.hpp. He then added printlines in the valid_values method to discover which line was failing. In our case that lead to us writing a test file stan_math/tests/foo_test.cpp:

#include <stan/math/mix/scal.hpp>
#include <gtest/gtest.h>

TEST(foo, bar) {
  double y = 0;
  double mu = 0;
  stan::math::fvar<stan::math::var> sigma = 1;
  stan::math::fvar<stan::math::var> lp = normal_lpdf(y, mu, sigma);
}

We can then run this test with ./runTests.py test/foo_test.cpp.

Next, we compiled this simplified test with debugging flags using make CC='clang++ -g' test/foo_test. I also edited the makefile to change O = 0 from O = 3 optimization level (which I seem to remember helping long ago). I think I also had an issue with make not detecting some relevant changes and not rebuilding, so issueing the generating clang++ commands to build and link seemed to help in that case.

Now we were set up to use lldb (since we’re using clang), which we did using lldb test/foo_test and then telling the resulting prompt to run. It then segfaulted and we were able to see the source code around the error and use up and down to navigate up and down the stack and print to print the values of variables.

Hope this helps someone (maybe myself ;)) in the future!
-Sean

1 Like