All the numbers above are without the profiler with -O3. There aren’t any special compiler flags. I think it works by just interrupting the process and writing down the stack trace X times per second (default 100 times).
N = 80, adjoint sensitivity, NaNs removed
8.74s
N = 80, adjoint sensitivity, NaN checks in place
10.59s
This is the file I used to do the benchmarking of fwd vs. reverse mode Jacobian calculations: https://gist.github.com/bbbales2/e3f790a3f5496c0115cd5c64d8daaa3a
It basically computes the Jacobian above in 10000 times for each fwd and rev modes and compares the time per run.
w/ profiler attached, w/ nan checks:
bbales2@frog:~/math-adjoint$LD_PRELOAD=/usr/lib/libprofiler.so CPUPROFILE=cpu_profile ./test_jacobian
Reverse mode: 6.23723e-05
Forward mode: 3.69096e-05
w/o profiler attached, w/ nan checks:
bbales2@frog:~/math-adjoint$ ./test_jacobian
Reverse mode: 6.21455e-05
Forward mode: 3.68801e-05
w/ profiler, w/o nan checks:
bbales2@frog:~/math-adjoint$ LD_PRELOAD=/usr/lib/libprofiler.so CPUPROFILE=cpu_profile ./test_jacobian
Reverse mode: 6.18991e-05
Forward mode: 2.63117e-05
w/o profiler, w/o nan checks:
bbales2@frog:~/math-adjoint$ ./test_jacobian
Reverse mode: 6.29211e-05
Forward mode: 2.61457e-05
I tried to collect a perf graph that showed the is_nans. Didn’t seem to work. Maybe I was running with -O1 the first time I did it? Anyway looks like it’s being inlined now so it’s not showing up in the stack traces.