CmdStan on ARM error

General
1

I am attempting to install CmdStan via CmdStanR on a AWS Graviton2 instance running Ubuntu 20.04. install_cmdstan() happily runs until trying to compile one of the examples, failing with:

/bin/bash: bin/stanc: cannot execute binary file: Exec format error
make: *** [make/program:53: examples/bernoulli/bernoulli.hpp] Error 126

The problem is that stanc was not properly compiled and cannot run:

~/.cmdstanr/cmdstan-2.24.1/bin$ ./stanc
-bash: ./stanc: cannot execute binary file: Exec format error

Is this a bug in how the compilation for stanc is working in a non-x86 environment? Is this supported?

2

stanc3 is currently downloaded as a binary. I think it’s a bit of a trick to build and distributing it as a binary was an easy solution.

If you can build your own version, you could probably use that. The repository is over here: https://github.com/stan-dev/stanc3

There are build directions there. @rybern are those up to date?

3

I was hoping the Nix build would be straightforward, but it fails with:

error: cannot coerce a set to a string, at /nix/store/h497gh2wc1zfv92928i7i5djrrfa76d6-nixpkgs-19.09/pkgs/build-support/ocaml/dune.nix:30:3

I am using the release/v2.24.1 branch. Is there a potentially easy fix, or is this a more complicated debugging endeavor?

4

Shoot, Nix shouldn’t be giving any trouble. I don’t recognize that error. I can look into it tomorrow morning. The non-Nix instructions should also work but I don’t use them myself.

5

Not sure. I think the reason we distribute the binaries is it can be a bit of a trick.

6

Thanks for taking a look. Unfortunately setting up dependencies via setup_dev_env.sh also fails:

[ERROR] The compilation of core failed at “/home/ubuntu/.opam/opam-init/hooks/sandbox.sh build jbuilder build -p core -j 31”.

Fatal error: exception (runtime-lib/runtime.ml.E “comparison failed”
((switch true) vs (switch false) (Loc src/config/discover.ml:205:15)))
Raised at file “src/configurator.ml”, line 511, characters 13-22
Called from file “src/config/discover.ml”, line 136, characters 2-1023

7

As a quick followup,

nix-shell --command "dune build"

appears to have been more successful than nix-build. However, when compiling my Stan program, the compiler runs for some time and eventually ends with a seg fault. I guess running Stan on ARM is not an easy feat. This would be great to investigate further given the low cost of Graviton2 and Stan’s new within-chain parallelization.

8

The easiest fix here is to use stanc2. This won’t give you the latest features, but that compiles from the c++ sources. I think setting STANC2=true In make local suffices.

9

Thanks. Unfortunately the motivation for doing this on ARM/Graviton2 was to take advantage of the large multi-core setup and reduce_sum. Since reduce_sum was introduced in Stan 2.23 and stanc2 ended with the 2.22 release, I am assuming it is not available there.

10

Oh…then that is not an option.

The other way to work around this is to transpile the Stan file on a different machine and copy over the hpp then…but I have never done that…

11

reduce_sum is not available in stanc2. stanc2 is there to insure that models developed before 2.22 will be able to be compiled as before; it is slated for deprecation.

12

interesting idea - you’d need to build the exe file directly from the command line (i.e., using the CmdStan makefile outside of cmdstanr) - at which point you’d have a .stan and .exe file where latter is newer than the former and so it would skip the compile. might work - but what a PITA!

13

I set up a test with two AWS instances, one ARM and one x86. nix-build did fail on ARM, so I’ll have to work on that, but nix-shell “dune build” worked as you said.

But, after building with Nix, I was not able to replicate the segfault in stanc3 on ARM. I’m betting that issues was downstream of stanc3, is that possible @asceles? Unfortunately that would eliminate the hpp-copying solution.

14

It’s certainly possible. I am not using stanc3 directly but via CmdStan/CmdStanR. FWIW, the exact same model file compiles and works well on x86. It could be some interaction between CmdStan/CmdStanR and stanc3 causing the issue though.

15

The test suite for stanc3 passed on ARM, so I suspect the issue is downstream.

(By the way, that issue with Nix should be merged to master soon)

16

I’m not sure what you mean by “interaction”.

stanc3 is a program which given a Stan program as an input file, parses the file and produces a C++ output file. e.g. stanc3 translates bernoulli.stan to bernoulli.hpp. that’s it. that C++ file is the same regardless of whatever platform is used to do the compilation, and is the same if you’re using CmdStan or any of the interfaces.

if you’ve got CmdStanR installed on AWS, then there should be a hidden folder $HOME/.cmdstanr - you’ll find the CmdStan install there - either $HOME/.cmstanr/cmdstan or $HOME/.cmdstanr/cmdstan-2.24.1 or similar. cd to this directory, and then try to compile from the command line. cf. example in the CmdStan manual here: 14 Parallelization | CmdStan User’s Guide

what exactly do you mean by x86 ? does it run multi-threaded?

17

I borrowed the stanc from cmdstanpy installation directory.
anaconda3/envs/my_env/bin/cmdstan/bin/stanc
copy to cmdstan-2.28.2/bin.

18

I ran into a similar issue, the command in Pre-complied cmdstan model 'cannot execute binary file' error - #11 by rok_cesnovar worked for me