Using cmdstan makefile from another directory

I am using Linux (Ubuntu 18.10), cmdstan 2.18.1, with the usual gcc and GNU make setup.

I am aware that one should use the provided makefile to compile programs as

cd cmdstan_location
make /some_other_path/model

but I am wondering if I could call it in the directory of the model file somehow, as it would make my workflow more convenient.

So far I tried

make -C cmdstan_location -f cmdstan_location/makefile model

but got an error, eg

$ make -C ~/src/cmdstan-2.18.1/ -f ~/src/cmdstan-2.18.1/makefile model
make: Entering directory '/home/tamas/src/cmdstan-2.18.1'
make: *** No rule to make target 'model'.  Stop.
make: Leaving directory '/home/tamas/src/cmdstan-2.18.1'

I’ve been working, for a while now, on a bash script that does this. Omitting the extra bits, these lines from this script file work on macOS 10.13.6.

cwd="$(pwd)"
cmdstan="$HOME/cmdstan"
make "$cwd/$1" -C "$cmdstan"
1 Like

Here are some instructions, that I did a while ago. (On some system there might be problems, not sure why.)

ps. Don’t have spaces in your path.

1 Like

Yes, that’s crucial. There’s no way to have make treat spaces properly.

I’m not sure if this will work with the current develop makefiles, but please report if it does work.

@syclik the following bash script works for me (macOS 10.13) with a fresh clone of cmdstan. It’s certainly a work in progress, but lines 8, 9, and 94 are what I posted above.

I can make the .hpp file in the right place, but I’m not able to make the .hpp.gch or binary in the right place. The above script didn’t work for me either. I would appreciate any tips you can offer. I’d be willing to patch the makefile if that would work but I’m not sure how to do it.

% make "$PWD"/examples/bernoulli/bernoulli -C /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/ 
make: Entering directory '/nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan'

--- Translating Stan model to C++ code ---
bin/stanc  /run/user/1000/tmp.ycQ61V1UzR/examples/bernoulli/bernoulli.stan --o=/run/user/1000/tmp.ycQ61V1UzR/examples/bernoulli/bernoulli.hpp
Model name=bernoulli_model
Input file=/run/user/1000/tmp.ycQ61V1UzR/examples/bernoulli/bernoulli.stan
Output file=/run/user/1000/tmp.ycQ61V1UzR/examples/bernoulli/bernoulli.hpp
Compiling pre-compiled header
g++ -Wall -I . -isystem stan/lib/stan_math/lib/eigen_3.3.3 -isystem stan/lib/stan_math/lib/boost_1.64.0 -isystem stan/lib/stan_math/lib/cvodes_2.9.0/include -std=c++1y -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -Wno-unused-function -Wno-uninitialized -I src -isystem stan/src -isystem stan/lib/stan_math/ -DFUSION_MAX_VECTOR_SIZE=12 -Wno-unused-local-typedefs -DEIGEN_NO_DEBUG -DNO_FPRINTF_OUTPUT -pipe  -c -O3 stan/src/stan/model/model_header.hpp -o stan/src/stan/model/model_header.hpp.gch
In file included from <command-line>:
/nix/store/dkm3gwl0xrx0wrw6zi5x3px3lpgjhlw4-glibc-2.32-dev/include/stdc-predef.h:1: fatal error: can’t create precompiled header stan/src/stan/model/model_header.hpp.gch: Read-only file system
    1 | /* Copyright (C) 1991-2020 Free Software Foundation, Inc.
      | 
compilation terminated.
make: *** [make/models:10: stan/src/stan/model/model_header.hpp.gch] Error 1
make: Leaving directory '/nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan'

% tree examples  
examples
└── bernoulli
    ├── bernoulli.data.R
    ├── bernoulli.hpp
    └── bernoulli.stan

1 directory, 3 files

% stanc --version                                                              
stanc version 2.17.1

You might be able to get it to work that way by providing the STAN make variable.

To be specific:

make "$PWD"/examples/bernoulli/bernoulli -C /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/ STAN= /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/

I didn’t try it out, so not certain it’ll work. Also, if it does, you might be able to put that variable in a shared make variables file so you don’t need to type it out repeatedly.

Thanks for replying. Specifying STAN= has the same result as above: it creates the .hpp but not the .hpp.gch or binary files.

% make "$PWD"/examples/bernoulli/bernoulli -C /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/ STAN=/nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/
make: Entering directory '/nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan'

--- Translating Stan model to C++ code ---
bin/stanc  /tmp/tmp.eoFIQnMDlR/examples/bernoulli/bernoulli.stan --o=/tmp/tmp.eoFIQnMDlR/examples/bernoulli/bernoulli.hpp
Model name=bernoulli_model
Input file=/tmp/tmp.eoFIQnMDlR/examples/bernoulli/bernoulli.stan
Output file=/tmp/tmp.eoFIQnMDlR/examples/bernoulli/bernoulli.hpp
Compiling pre-compiled header
g++ -Wall -I . -isystem /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/lib/stan_math/lib/eigen_3.3.3 -isystem /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/lib/stan_math/lib/boost_1.64.0 -isystem /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/lib/stan_math/lib/cvodes_2.9.0/include -std=c++1y -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -Wno-unused-function -Wno-uninitialized -I src -isystem /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/src -isystem /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/lib/stan_math/ -DFUSION_MAX_VECTOR_SIZE=12 -Wno-unused-local-typedefs -DEIGEN_NO_DEBUG -DNO_FPRINTF_OUTPUT -pipe  -c -O3 /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/src/stan/model/model_header.hpp -o /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/src/stan/model/model_header.hpp.gch
In file included from <command-line>:
/nix/store/dkm3gwl0xrx0wrw6zi5x3px3lpgjhlw4-glibc-2.32-dev/include/stdc-predef.h:1: fatal error: can’t create precompiled header /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/src/stan/model/model_header.hpp.gch: Read-only file system
    1 | /* Copyright (C) 1991-2020 Free Software Foundation, Inc.
      | 
compilation terminated.
make: *** [make/models:10: /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/src/stan/model/model_header.hpp.gch] Error 1
make: Leaving directory '/nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan'

Thanks for including the full error message.

It doesn’t look like the variable was set. Or maybe 2.17.1 had a bug that was fixed; I was looking at the current source.

Let me check the old version to see if it’ll work.

It does look like for v2.17.1 the procompiled header location should be based on the STAN variable.

Not sure what the best way to figure out why the variable isn’t getting propagated. If you’re pretty strong with Make, I can try to help out. If not, it’s not the most fun thing to debug.

Actually, I misread the last error… it says it’s a read-only file system. Are you able to write to that location?

Are you able to write to that location?

That location is in the /nix/store/ filesystem, which is where the Nix package manager stores package files. It is read-only for users. This is the same problem described in this NixOS forum post and this Tweag/economics issue.

I believe Make is trying to create the gch file in opt/cmdstan/stan/src/stan/model/. I would like it to instead create that file (and the subsequent binary) in a user-specified directory such as the current directory.

MODEL_HEADER := $(STAN)src/stan/model/model_header.hpp
MODEL_PCH := $(MODEL_HEADER:.hpp=.hpp.gch)
CMDSTAN_MAIN := src/cmdstan/main.cpp

$(MODEL_PCH) : $(MODEL_HEADER)
	@echo 'Compiling pre-compiled header'
	$(COMPILE.cc) -O$O $(MODEL_HEADER) -o $@

.PRECIOUS: %.hpp %.o
$(patsubst %.stan,%,$(wildcard $(addsuffix .stan,$(MAKECMDGOALS)))) : %$(EXE) : %.hpp %.stan bin/stanc$(EXE) bin/stansummary$(EXE) bin/diagnose$(EXE) $(LIBCVODES) $(MODEL_PCH)
	@echo ''
	@echo '--- Linking C++ model ---'
ifneq (,$(findstring allow_undefined,$(STANCFLAGS)))
	$(LINK.cc) $(if $(filter $(CC_TYPE),clang++),-include-pch $(MODEL_PCH)) -O$O $(OUTPUT_OPTION) $(CMDSTAN_MAIN) -include $< -include $(USER_HEADER) $(LIBCVODES))
else
	$(LINK.cc) $(if $(filter $(CC_TYPE),clang++),-include-pch $(MODEL_PCH)) -O$O $(OUTPUT_OPTION) $(CMDSTAN_MAIN) -include $< $(LIBCVODES)
endif

Thanks. Unfortunately, that’s not how precompiled headers work.

You might be able to disable the precompiled header completely.

Did you already build the runtime libraries before copying into the directory? If so, maybe there’s a way to generate the precompiled library file before copying it there.

Did you already build the runtime libraries before copying into the directory? If so, maybe there’s a way to generate the precompiled library file before copying it there.

Is it necessary for that pch to be generated dynamically for each .stan file, or could it be built once when cmdstan itself is built? In the bernoulli example the contents are:

// % cat /nix/store/w0cz95m1482l8p2a68s3gs35aq80wmzs-cmdstan-2.17.1/opt/cmdstan/stan/src/stan/model/model_header.hpp
#ifndef STAN_MODEL_MODEL_HEADER_HPP
#define STAN_MODEL_MODEL_HEADER_HPP

#include <stan/math.hpp>

#include <stan/io/cmd_line.hpp>
#include <stan/io/dump.hpp>
#include <stan/io/program_reader.hpp>
#include <stan/io/reader.hpp>
#include <stan/io/writer.hpp>

#include <stan/lang/rethrow_located.hpp>
#include <stan/model/prob_grad.hpp>
#include <stan/model/indexing.hpp>
#include <stan/services/util/create_rng.hpp>

#include <boost/exception/all.hpp>
#include <boost/random/additive_combine.hpp>
#include <boost/random/linear_congruential.hpp>

#include <cmath>
#include <cstddef>
#include <fstream>
#include <iostream>
#include <sstream>
#include <stdexcept>
#include <utility>
#include <vector>

#endif

I patched make/models to remove the PCH files as shown below, which allowed the model to compile.

diff --git a/make/models b/make/models
index 56b05c76..90b99ae3 100644
--- a/make/models
+++ b/make/models
@@ -2,19 +2,15 @@
 # Models (to be passed through stanc)
 ##
 MODEL_HEADER := $(STAN)src/stan/model/model_header.hpp
-MODEL_PCH := $(MODEL_HEADER:.hpp=.hpp.gch)
 CMDSTAN_MAIN := src/cmdstan/main.cpp

-$(MODEL_PCH) : $(MODEL_HEADER)
-	@echo 'Compiling pre-compiled header'
-	$(COMPILE.cc) -O$O $(MODEL_HEADER) -o $@

 .PRECIOUS: %.hpp %.o
-$(patsubst %.stan,%,$(wildcard $(addsuffix .stan,$(MAKECMDGOALS)))) : %$(EXE) : %.hpp %.stan bin/stanc$(EXE) bin/stansummary$(EXE) bin/diagnose$(EXE) $(LIBCVODES) $(MODEL_PCH)
+$(patsubst %.stan,%,$(wildcard $(addsuffix .stan,$(MAKECMDGOALS)))) : %$(EXE) : %.hpp %.stan bin/stanc$(EXE) bin/stansummary$(EXE) bin/diagnose$(EXE) $(LIBCVODES)
 	@echo ''
 	@echo '--- Linking C++ model ---'
 ifneq (,$(findstring allow_undefined,$(STANCFLAGS)))
-	$(LINK.cc) $(if $(filter $(CC_TYPE),clang++),-include-pch $(MODEL_PCH)) -O$O $(OUTPUT_OPTION) $(CMDSTAN_MAIN) -include $< -include $(USER_HEADER) $(LIBCVODES))
+	$(LINK.cc) -O$O $(OUTPUT_OPTION) $(CMDSTAN_MAIN) -include $< -include $(USER_HEADER) $(LIBCVODES))
 else
-	$(LINK.cc) $(if $(filter $(CC_TYPE),clang++),-include-pch $(MODEL_PCH)) -O$O $(OUTPUT_OPTION) $(CMDSTAN_MAIN) -include $< $(LIBCVODES)
+	$(LINK.cc) -O$O $(OUTPUT_OPTION) $(CMDSTAN_MAIN) -include $< $(LIBCVODES)
 endif

 .PRECIOUS: %.hpp

I’m not sure how much effect removing the PCH will have on model compilation time. Compilation can take a while and if that’s a significant part of the time it would be nice to cache it one way or another.

PCH files reduce the compilation time for about 50% in latest versions of cmdstan. Its probably less in 2.17 that did not have the precompiled main.o object file and CRTP.

In latest Cmdstan versons you can disable precompiled headers by setting PRECOMPILED_HEADERS=false. This is unfortunately not available in 2.17.

2 Likes

On 2.25.0, it works without any help. Thanks for the suggestions.

% make "$PWD"/examples/bernoulli/bernoulli -C result/opt/cmdstan/ 
make: Entering directory '/nix/store/iy76xsp7cz0l79pv0fwq75vfry6pjfh4-cmdstan-2.25.0/opt/cmdstan'

--- Translating Stan model to C++ code ---
bin/stanc  --o=/run/user/1000/tmp.19nkTDoaqd/examples/bernoulli/bernoulli.hpp /run/user/1000/tmp.19nkTDoaqd/examples/bernoulli/bernoulli.stan

--- Compiling, linking C++ code ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.7 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.2.0/include    -DBOOST_DISABLE_ASSERTS        -c -Wno-ignored-attributes   -x c++ -o /run/user/1000/tmp.19nkTDoaqd/examples/bernoulli/bernoulli.o /run/user/1000/tmp.19nkTDoaqd/examples/bernoulli/bernoulli.hpp
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.7 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.2.0/include    -DBOOST_DISABLE_ASSERTS              -Wl,-L,"/nix/store/iy76xsp7cz0l79pv0fwq75vfry6pjfh4-cmdstan-2.25.0/opt/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/nix/store/iy76xsp7cz0l79pv0fwq75vfry6pjfh4-cmdstan-2.25.0/opt/cmdstan/stan/lib/stan_math/lib/tbb"      /run/user/1000/tmp.19nkTDoaqd/examples/bernoulli/bernoulli.o src/cmdstan/main.o         stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_kinsol.a  stan/lib/stan_math/lib/tbb/libtbb.so.2 -o /run/user/1000/tmp.19nkTDoaqd/examples/bernoulli/bernoulli
rm -f /run/user/1000/tmp.19nkTDoaqd/examples/bernoulli/bernoulli.o
make: Leaving directory '/nix/store/iy76xsp7cz0l79pv0fwq75vfry6pjfh4-cmdstan-2.25.0/opt/cmdstan'