Using opencl with cmdstanpy

In order to use OpenCL with cmdstanpy, I know that the model has to be compiled with the OPENCL_DEVICE_ID and OPENCL_PLATFORM_ID parameters set. How does one find these parameters?

1 Like

This section of the manual covers the basic requirements for getting OpenCL setup, including the clinfo utility: 14 Parallelization | CmdStan User’s Guide

In most cases the default 0 for the platform and device ID is fine, but you can verify by running (from the terminal):

clinfo -l

I’m on Centos and when I run yum --version OpenCL I can see my OpenCL version but I can’t run the command clinfo -l. Is there any way to use the pyopencl library on python to get device and platform id’s?

Also, when I compile my model on cmdstanpy

model = CmdStanModel(
    stan_file = "normal_exponential.stan",
    cpp_options = {'STAN_THREADS': True,
                   'STAN_OPENCL': True,
                   'OPENCL_PLATFORM_ID':0,
                   'OPENCL_DEVICE_ID':0
                  },
    compile = "force"
)

I get the following warning message, which I’m not sure how to interpret so I’d appreciate guidance on this!

10:15:52 - cmdstanpy - INFO - compiling stan file /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan to exe file /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential
10:16:36 - cmdstanpy - INFO - compiled model executable: /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential
10:16:36 - cmdstanpy - WARNING - Stan compiler has produced 1 warnings:
10:16:36 - cmdstanpy - WARNING - 
--- Translating Stan model to C++ code ---
bin/stanc --use-opencl --o=/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.hpp /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan
Warning in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 6, column 2: Declaration
    of arrays by placing brackets after a variable name is deprecated and
    will be removed in Stan 2.32.0. Instead use the array keyword before the
    type. This can be changed automatically using the auto-format flag to
    stanc

--- Compiling, linking C++ code ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.78.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes        -c -Wno-ignored-attributes   -x c++ -o /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.o /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.hpp
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.78.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes              -Wl,-L,"/home/chrisliao/.cmdstan/cmdstan-2.30.1/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/chrisliao/.cmdstan/cmdstan-2.30.1/stan/lib/stan_math/lib/tbb"      /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.o src/cmdstan/main_threads_opencl.o      -lOpenCL  -Wl,-L,"/home/chrisliao/.cmdstan/cmdstan-2.30.1/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/chrisliao/.cmdstan/cmdstan-2.30.1/stan/lib/stan_math/lib/tbb"   stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_kinsol.a  stan/lib/stan_math/lib/tbb/libtbb.so.2 -o /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential
rm -f /home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.o

but I can’t run the command clinfo -l.

The manual section I posted above says:

On Linux, clinfo can typically be installed with the default package manager

For Centos, this would be:

yum install clinfo

There are no errors in your output, only a warning about your Stan syntax. The warning says:

Warning in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 6, column 2: 
Declaration of arrays by placing brackets after a variable name is deprecated and
    will be removed in Stan 2.32.0. Instead use the array keyword before the
    type. This can be changed automatically using the auto-format flag to
    stanc

Which is referring to the use of the “old” style of declaring arrays. More background on the recommended way to declare arrays is: 5.6 Array data types | Stan Reference Manual

Got it, thank you! Also, is the sampling process for cmdstanpy when OpenCL is integrated versus not integrated? My sampling process runs without issue when OpenCL is not enabled, but when I use OpenCL, I get sampling issues like the ones below (some of which don’t make sense).

	Exception: exponential_lpdf(OpenCL): Random variable[1073359579, 1797372518] = 0.158811, but it must be nonnegative! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 20, column 2 to column 42)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[450, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[3, 0] = inf, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[1, 0] = inf, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Random variable[1074926712, 923931041] = 0, but it must be nonnegative! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[65, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Random variable[-1054495369, 1171463445] = 0, but it must be nonnegative! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 20, column 2 to column 42)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[448, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[64, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[503, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[517, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[96, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: normal_lpdf(OpenCL): Location parameter[195, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[577, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[195, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[183, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Scale parameter[-1073327113, -1340029796] = 0, but it must be positive! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[520, 0] = inf, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: normal_lpdf(OpenCL): Location parameter[5928, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[577, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[577, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[1070884330, 762234075] = 0.732286, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: normal_lpdf(OpenCL): Location parameter[5527, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[512, 0] = inf, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: normal_lpdf(OpenCL): Location parameter[7324, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[1068825436, -1048612554] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 20, column 2 to column 42)
	Exception: normal_lpdf(OpenCL): Location parameter[577, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[7324, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[547, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: normal_lpdf(OpenCL): Location parameter[7324, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[7324, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[-1070697441, 0] = -2.791, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: normal_lpdf(OpenCL): Location parameter[-1073327113, -1340029796] = 0, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: exponential_lpdf(OpenCL): Random variable[-1075693246, -1903345647] = -1.30035, but it must be nonnegative! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 20, column 2 to column 42)
	Exception: exponential_lpdf(OpenCL): Inverse scale parameter[65, 0] = 0, but it must be positive finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 22, column 2 to column 40)
	Exception: normal_lpdf(OpenCL): Location parameter[7324, 0] = -inf, but it must be finite! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 23, column 2 to column 50)
	Exception: exponential_lpdf(OpenCL): Random variable[1069959489, 388503752] = 0.378754, but it must be nonnegative! (in '/home/chrisliao/7mde_bayes/normal_exponential_3/normal_exponential.stan', line 20, column 2 to column 42)