Problem with matrix multiplication in rstan (Linux, intel compiler)

Hello,

I am having problems running a model with rstan on linux, which works on OS-X or Windows 10.

The model is

data {
 int N;
 int D;
 matrix[N,D] y;
}

parameters {
  vector[D] mu;
  vector<lower = 0>[D] sigma;  
  cholesky_factor_corr[D] L_Omega;
}

transformed parameters {
  cov_matrix[D] Sigma;  
  Sigma = quad_form_diag(L_Omega * L_Omega', sigma);
}

model {
  L_Omega ~ lkj_corr_cholesky(2);
  mu ~ normal(0,3);
  sigma ~ normal(0,1);
  for (n in 1:N)
    target += multi_normal_lpdf(y[n,] | mu, Sigma);
}

When I try to run this model, I get the following error:

Exception: validate transformed params: Sigma is not symmetric. Sigma[3,5] = 0.158665, but Sigma[5,3] = 0.168324 (in ‘model70a454bc4e31_EX’ at line 14)
[1] “Error in sampler$call_sampler(args_list[[i]]) : "
[2] " Exception: validate transformed params: Sigma is not symmetric. Sigma[3,5] = 0.158665, but Sigma[5,3] = 0.168324 (in ‘model70a454bc4e31_EX’ at line 14)”
error occurred during calling the sampler; sampling not done

To be sure that the problem with matrix multiplication is not a more general problem, I ran the 8-Schools example, and this worked just fine. The model posted above also works on OS-X and Windows 10 machines.

I am wondering if the problem has to do with the libraries used for matrix multiplication. One factor that maybe contributes to my problem is that I have to use the intel compiler which is available on the cluster I am working on.

Any help is very much appreciated!

Guido

R sessionInfo

R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS release 6.9 (Final)

Matrix products: default
BLAS/LAPACK: /cluster/software/VERSIONS/intel-2017.2/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so

locale:
[1] LC_CTYPE=C LC_NUMERIC=C
[3] LC_TIME=en_US.iso885915 LC_COLLATE=en_US.iso885915
[5] LC_MONETARY=en_US.iso885915 LC_MESSAGES=en_US.iso885915
[7] LC_PAPER=en_US.iso885915 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.iso885915 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] rstan_2.17.3 StanHeaders_2.17.2 ggplot2_2.2.1 data.table_1.11.2

loaded via a namespace (and not attached):
[1] Rcpp_0.12.16 codetools_0.2-15 grid_3.4.4 plyr_1.8.4
[5] gtable_0.2.0 stats4_3.4.4 scales_0.5.0 pillar_1.1.0
[9] rlang_0.1.6 lazyeval_0.2.1 tools_3.4.4 munsell_0.4.3
[13] compiler_3.4.4 inline_0.3.14 colorspace_1.3-2 gridExtra_2.3
[17] tibble_1.4.2

MAKEVARS FILE

MKLROOT = /cluster/software/VERSIONS/intel-2017.2/mkl/

CFLAGS += -I $(MKLROOT)/include -DEIGEN_USE_MKL_ALL

LDLIBS += -L$(MKLROOT)/lib/intel64 -lmkl_intel_lp64

LDLIBS += -lmkl_core -lmkl_sequential -lpthread -lm

CXXFLAGS=-O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function -Wno-macro-redefined

The problem persists if I only the last line (CXXFLAGS …) is in .R/MAKEVARS.

WARNINGS WHEN INSTALLING rstan

A warning like the follow or similar frequently occurs during installation of rstan

In file included from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/spirit/include/phoenix_limits.hpp(11),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/spirit/home/support/meta_compiler.hpp(16),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/spirit/home/qi/meta_compiler.hpp(14),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/spirit/home/qi/action/action.hpp(14),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/spirit/home/qi/action.hpp(14),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/spirit/home/qi.hpp(14),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/spirit/include/qi.hpp(16),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/lang/grammars/whitespace_grammar.hpp(4),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/lang/grammars/program_grammar.hpp(6),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/lang/parser.hpp(5),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/lang/compiler.hpp(7),
from stanc.cpp(22):
/cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/phoenix/core/limits.hpp(44): warning #47: incompatible redefinition of macro “BOOST_PHOENIX_NO_VARIADIC_EXPRESSION”
define BOOST_PHOENIX_NO_VARIADIC_EXPRESSION
^

WARNINGS WHEN COMPILING THE MODEL

In file included from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/config/compiler/intel.hpp(44),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/config.hpp(39),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/math/tools/config.hpp(13),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/core/var.hpp(7),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/core/gevv_vvv_vari.hpp(5),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/core.hpp(12),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math/rev/mat.hpp(4),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/stan/math.hpp(4),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/model/model_header.hpp(4),
from file70a433621e00.cpp(8):
/cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/config/compiler/gcc.hpp(186): warning #47: incompatible redefinition of macro “BOOST_NO_CXX11_RVALUE_REFERENCES”
define BOOST_NO_CXX11_RVALUE_REFERENCES
^

In file included from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/services/optimize/bfgs.hpp(11),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/stan_fit.hpp(36),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/rstaninc.hpp(3),
from file70a433621e00.cpp(507):
/cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/StanHeaders/include/src/stan/optimization/bfgs.hpp(113): warning #858: type qualifier on return type is meaningless
const size_t iter_num() const { return _itNum; }
^

In file included from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/stan_fit.hpp(56),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/rstaninc.hpp(3),
from file70a433621e00.cpp(507):
/cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/sum_values.hpp(41): warning #858: type qualifier on return type is meaningless
const size_t called() const {
^

In file included from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/stan_fit.hpp(56),
from /cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/rstaninc.hpp(3),
from file70a433621e00.cpp(507):
/cluster/home/guidopb/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/sum_values.hpp(45): warning #858: type qualifier on return type is meaningless
const size_t recorded() const {
^

1 Like

I’m not sure if this is the problem, but you need to use some extra CFLAGS when you use the intel compiler. They are on pp. 83-84 of the cmdstan manual.

May I ask why you need to use the Intel compiler? In my experience GCC produces faster Stan programs.

You want to be using

Sigma = quad_form_diag(multiply_lower_tri_self_transpose(L_Omega), sigma);

Thanks,

works.

Do you know why my original code worked on windows and mac, but not on linux with the intel compiler?

On the cluster I am working on the intel compiler is the default compiler. I see if I can figure out how to use GCC.

No, but I would also make sure icpc is not being passed the “fast math” switch or whatever it is called that does math wrong.