Reduce_sum_static fails to compile with SIGSEGV error - PyStan3

Modeling
#6

That is a compilation error.

I’m not sure what is the current situation with the verbosity settings.

Can you try to compile the model with CmdStan (CmdStanPy) and see if there is an error msg.

#7

I have an error while downloading CmdStan to the cluster using install_cmdstan():

INFO:cmdstanpy:stan/lib/stan_math/stan/math/prim/prob/neg_binomial_2_log_glm_lpmf.hpp:139:17: error: expected ?;? before ?theta_tmp? WARNING:cmdstanpy:CmdStan installation failed

This is the last lines.

Is there a way to download not the latest version (2-24.0), but 2-23.0 with install_cmdstan()? Maybe this causes the problem.

I also tried uploading cmdstan-2.23.0.tar.gz directly to the cluster, but cmdstanpy didn’t recognize the binaries.

#8

But when I compile it on my local machine using CmdStanPy, it compiles without errors.

#9

I don’t know if that helps, but PyStan3 fails to compile models that were complied fine with PyStan2.19.

#10

Check the version of the c++ compiler on the cluster in that case.

#11

Checked that on the cluster:

>g++ --version
>g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)

On my local machine it’s

g++ --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I know CmdStan documentation suggests using 4.9.3 or later, but it works fine on my local machine.

#12

Your local compiler is clang 9.0, which is why ot works locally.

I can confirm it will definitely not work with 4.8.5.

#13

ok, that’s good to know about clang! Thanks!
I asked my system admins if C++ compiler can be updated, so let’s hope for the best.
Is this also why PyStan3 models don’t compile?

#14

On the cluster you will not be able to compile with g++ 4.8.5 with any Stan interface, so I would imagine so. All interfaces have the same backend that requires g++ 4.9.3 or clang 5.0+ (we officially say 6 because that is what we test).

#15

Hmm are you sure about that? I run models just fine with pystan2.19 on this cluster…
Also, is there a way to easily change g++ to clang? Maybe cluster’s clang version is ok.

#16

On the cluster run: clang++ --version to see if it exists.

pystan2.19 uses the older version of the Math backend that could be used with older compiler pre-4.9.3.

#17

clang++: command not found… so now it’s all in the hands of the admins to update g++. So many setbacks!
And it makes sense about pystan2.19, thanks!

#18

conda has its own compilers too (conda install possible), but I recommend following the official route on clusters.

#19

Thanks! Btw, I compiled my PyStan3 model in the command line of my local machine and it still fails with Segmentation fault: 11. CmdStanPy still compiles fine, but multithreading doesn’t work there…

#20

I can reproduce the segfault on Ubuntu with gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0.

I will create an issue for this. Even if there would be something wrong, things should never segfault.

#21

Thanks!
UPDATE - @rok_cesnovar @ahartikainen I updated g++ to 4.9.3 and cmdstan2-24 was successfully build. Thank you so much.

#22

I tried to reduce the example and noticed that compiling model without data causes it to segfault

This segfaults

import stan
import numpy as np
import logging

logging.basicConfig(level=logging.DEBUG)

program_code = """
data {
    int W;
    int N;
    int AAA;
    int BBB;
    vector[BBB] U[N,W];
}
parameters {
    vector[AAA] X[N,W];
    matrix[AAA, AAA] A;
    matrix[AAA, BBB] B;
}
model {
    to_vector(A) ~ normal(0,1);
    to_vector(B) ~ normal(0,1);
    for (w in 1:W) {
        for (n in 1:N) {
            if (w == 1) {
                X[n,w] ~ normal(0,1);
            }
            else {
                X[n,w] ~  normal((A * X[n,w] + B * U[n,w-1]), 1);
            }
        }
    }
}"""

print("Starting the building process")
posterior = stan.build(program_code, data={})
print("Done")

This doesn’t crash

import stan
import numpy as np
import logging

logging.basicConfig(level=logging.DEBUG)

program_code = """
data {
    int W;
    int N;
    int AAA;
    int BBB;
    vector[BBB] U[N,W];
}
parameters {
    vector[AAA] X[N,W];
    matrix[AAA, AAA] A;
    matrix[AAA, BBB] B;
}
model {
    to_vector(A) ~ normal(0,1);
    to_vector(B) ~ normal(0,1);
    for (w in 1:W) {
        for (n in 1:N) {
            if (w == 1) {
                X[n,w] ~ normal(0,1);
            }
            else {
                X[n,w] ~  normal((A * X[n,w] + B * U[n,w-1]), 1);
            }
        }
    }
}"""

print("Starting the building process")
posterior = stan.build(program_code, data={
    "W" : 3,
    "N" : 2,
    "AAA" : 4,
    "BBB" : 5,
    "U" : np.random.randn(2, 3, 5),
})
print("Done")

Still need to figure out the minimal example which fails.

#23

Added two issues

empty data

this model

#24

Thanks for addressing this! Does it mean that something is wrong with my model? (I bet many things are wrong with my model.)

#25

segfault is fault in our end.

But, there might be something wrong with your data?

E.g. is end used in the function? Is all parameters used in the model / function for user function?