Cmdstanpy: Not working with google colab GPU

I want to run cmdstan using the GPU of google colab.

While it works on the CPU, it cannot be confirmed to work on the GPU…


cpp_options = {
                #'STAN_THREADS'       : True , 
                #'STAN_CPP_OPTIMS'    : True, 
                'STAN_OPENCL'        : True ,
                'OPENCL_PLATFORM_ID' : 0    ,
                'OPENCL_DEVICE_ID'   : 0}

sm = cmdstanpy.CmdStanModel(stan_file="/content/gdrive/MyDrive/colab/stan_code.stan", cpp_options= cpp_options)
fit = sm.sample(data=df, chains=2, iter_warmup=500,iter_sampling=500, show_progress=True)

---- output ----

Chain 1 - done: 0%| | 1/1000 [07:35<126:19:31, 455.23s/it]
Chain 2 - warmup: 0%| | 0/1 [07:36<?, ?it/s]


↑It’s freezing.

Hi @kyo219 - we have had reports in the past of the show_progress function acting strangely in notebooks. It’s all been re-written in 1.0 which should be out soon (or if you want to do a pip install from github).

That said, if it’s working on the CPU I suspect it is another issue.

@mitzimorris - I know you’ve used cmdstanpy in Google collab. Is the issue that cmdstan itself would need to be rebuilt with these flags, or could it be an issue with external dependencies? Not sure what opencl requires

I dont have any familiartiy with Google Colab so not sure I can help a ton but can try. The OpenCL dependencies that are required are listed here: 14 Parallelization | CmdStan User’s Guide

thank you guys.

when I try

sudo apt install nvidia-cuda-toolkit

then, I got this error message.

Preparing to unpack .../48-nvidia-cuda-dev_9.1.85-3ubuntu1_amd64.deb ...
dpkg: error processing archive /tmp/apt-dpkg-install-1gv7Ba/48-nvidia-cuda-dev_9.1.85-3ubuntu1_amd64.deb (--unpack): trying to overwrite '/usr/include/cublas.h', which is
also in package libcublas-dev 10.2.1.243-1dpkg-deb: error: paste subprocess was killed by signal(Broken pipe)

my environment

/content# clinfo -l
Platform #0: NVIDIA CUDA
 `-- Device #0: Tesla P100-PCIE-16GB
!nvidia-smi

Thu Oct 14 10:39:31 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.74       Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   32C    P0    27W / 250W |      0MiB / 16280MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

!nvcc --version


nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0

I also try cmdstanr in Google colab.

I got the same problem with cmdstanpy.

install.packages("cmdstanr", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))
library(cmdstanr)
install_cmdstan(cores=4, overwrite = TRUE)
sm <- cmdstan_model("/stan_code.stan",
                        cpp_options = list(stan_opencl = TRUE))

its done

But . There is a problem with the sampling.

fit <- sm$sample(data = data, chains = 4, parallel_chains = 4,
                        opencl_ids = c(0, 0),  refresh = 0)
Running MCMC with 4 parallel chains...

Chain 1 Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:

Chain 1 Exception: stan_code_model_namespace::log_prob: o_Tau is not symmetric. o_Tau[1,2] = inf, but o_Tau[2,1] = inf (in '/tmp/RtmpEBdlH8/model-4d182d8a1c.stan', line 43, column 8 to column 40)

Chain 1 If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,

Chain 1 but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

.....

↑ freeze