Reduce_sum_static fails to compile with SIGSEGV error - PyStan3

Modeling
#7

I have an error while downloading CmdStan to the cluster using install_cmdstan():

INFO:cmdstanpy:stan/lib/stan_math/stan/math/prim/prob/neg_binomial_2_log_glm_lpmf.hpp:139:17: error: expected ?;? before ?theta_tmp? WARNING:cmdstanpy:CmdStan installation failed

This is the last lines.

Is there a way to download not the latest version (2-24.0), but 2-23.0 with install_cmdstan()? Maybe this causes the problem.

I also tried uploading cmdstan-2.23.0.tar.gz directly to the cluster, but cmdstanpy didn’t recognize the binaries.

#8

But when I compile it on my local machine using CmdStanPy, it compiles without errors.

#9

I don’t know if that helps, but PyStan3 fails to compile models that were complied fine with PyStan2.19.

#10

Check the version of the c++ compiler on the cluster in that case.

#11

Checked that on the cluster:

>g++ --version
>g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)

On my local machine it’s

g++ --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I know CmdStan documentation suggests using 4.9.3 or later, but it works fine on my local machine.

#12

Your local compiler is clang 9.0, which is why ot works locally.

I can confirm it will definitely not work with 4.8.5.

#13

ok, that’s good to know about clang! Thanks!
I asked my system admins if C++ compiler can be updated, so let’s hope for the best.
Is this also why PyStan3 models don’t compile?

#14

On the cluster you will not be able to compile with g++ 4.8.5 with any Stan interface, so I would imagine so. All interfaces have the same backend that requires g++ 4.9.3 or clang 5.0+ (we officially say 6 because that is what we test).

#15

Hmm are you sure about that? I run models just fine with pystan2.19 on this cluster…
Also, is there a way to easily change g++ to clang? Maybe cluster’s clang version is ok.

#16

On the cluster run: clang++ --version to see if it exists.

pystan2.19 uses the older version of the Math backend that could be used with older compiler pre-4.9.3.

#17

clang++: command not found… so now it’s all in the hands of the admins to update g++. So many setbacks!
And it makes sense about pystan2.19, thanks!

#18

conda has its own compilers too (conda install possible), but I recommend following the official route on clusters.

#19

Thanks! Btw, I compiled my PyStan3 model in the command line of my local machine and it still fails with Segmentation fault: 11. CmdStanPy still compiles fine, but multithreading doesn’t work there…

#20

I can reproduce the segfault on Ubuntu with gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0.

I will create an issue for this. Even if there would be something wrong, things should never segfault.

#21

Thanks!
UPDATE - @rok_cesnovar @ahartikainen I updated g++ to 4.9.3 and cmdstan2-24 was successfully build. Thank you so much.

#22

I tried to reduce the example and noticed that compiling model without data causes it to segfault

This segfaults

import stan
import numpy as np
import logging

logging.basicConfig(level=logging.DEBUG)

program_code = """
data {
    int W;
    int N;
    int AAA;
    int BBB;
    vector[BBB] U[N,W];
}
parameters {
    vector[AAA] X[N,W];
    matrix[AAA, AAA] A;
    matrix[AAA, BBB] B;
}
model {
    to_vector(A) ~ normal(0,1);
    to_vector(B) ~ normal(0,1);
    for (w in 1:W) {
        for (n in 1:N) {
            if (w == 1) {
                X[n,w] ~ normal(0,1);
            }
            else {
                X[n,w] ~  normal((A * X[n,w] + B * U[n,w-1]), 1);
            }
        }
    }
}"""

print("Starting the building process")
posterior = stan.build(program_code, data={})
print("Done")

This doesn’t crash

import stan
import numpy as np
import logging

logging.basicConfig(level=logging.DEBUG)

program_code = """
data {
    int W;
    int N;
    int AAA;
    int BBB;
    vector[BBB] U[N,W];
}
parameters {
    vector[AAA] X[N,W];
    matrix[AAA, AAA] A;
    matrix[AAA, BBB] B;
}
model {
    to_vector(A) ~ normal(0,1);
    to_vector(B) ~ normal(0,1);
    for (w in 1:W) {
        for (n in 1:N) {
            if (w == 1) {
                X[n,w] ~ normal(0,1);
            }
            else {
                X[n,w] ~  normal((A * X[n,w] + B * U[n,w-1]), 1);
            }
        }
    }
}"""

print("Starting the building process")
posterior = stan.build(program_code, data={
    "W" : 3,
    "N" : 2,
    "AAA" : 4,
    "BBB" : 5,
    "U" : np.random.randn(2, 3, 5),
})
print("Done")

Still need to figure out the minimal example which fails.

#23

Added two issues

empty data

this model

#24

Thanks for addressing this! Does it mean that something is wrong with my model? (I bet many things are wrong with my model.)

#25

segfault is fault in our end.

But, there might be something wrong with your data?

E.g. is end used in the function? Is all parameters used in the model / function for user function?

#26

Nope, I just use start as a participant index (the idea behind this design was to run all Ns in parallel). Hope this is implemented correctly.

Yes. I don’t think there are unused parameters. There are missing parameters that Stan imputes and that are used for expressions like alpha_rdm_pr[n,w] ~ normal(C[1,] * X[n,w],sigma_r);