GPU setup on AWS EC2 with Docker

Hi there,

I’m trying to set up an AWS G4 instance to get going using a GPU. At this point I’m just trying to get the Hello World bernoulli in the docs to run using CmdStanPy in our Docker container.

I first did:

apt-get update && apt-get install ocl-icd-opencl-dev nvidia-cuda-toolkit clinfo

then tried to fit the model:


import os
from cmdstanpy import cmdstan_path, CmdStanModel

bernoulli_stan = os.path.join(cmdstan_path(), 'examples', 'bernoulli', 'bernoulli.stan')

bernoulli_model = CmdStanModel(
    stan_file=bernoulli_stan,
    cpp_options={"STAN_OPENCL": True},
    )

bernoulli_data = os.path.join(
    cmdstan_path(), 'examples', 'bernoulli', 'bernoulli.data.json'
    )

bern_fit = bernoulli_model.sample(
    data=bernoulli_data,
    output_dir='/opt'
    )

but hit the following in stdout:

opencl_context: clGetPlatformIDs CL_PLATFORM_NOT_FOUND_KHR: Unknown error -1001

After a little research on the GPU threads here I started following instructions here.

I’m still stumped, though. Not sure if I missed some crucial documentation somewhere or if there are other tricks I need to be aware of, but the result of calling clinfo is:

Number of platforms                               0

The result of cat /etc/OpenCL/vendors/* is:

libnvidia-opencl.so.1

so am not exactly sure how to proceed with step 3 in the doc I linked to. Would appreciate any assistance or links to docs, or a redirect if I’m entirely misguided here.

Thanks for your help!

I think the only thing missing is a reboot of the instance.

The driver is installed with the cuda-toolkit so that should be good.

1 Like

I apologize, it seems that installing the driver separately is required. So the minimal instructions are

sudo apt-get -y update
sudo apt-get install -y nvidia-driver-460 nvidia-cuda-toolkit clinfo

Someone seeing this at a later point should replaced 460 with a larger number if available.

2 Likes

@rok_cesnovar thanks for the suggestions! Still having trouble here, though. I think it is perhaps made a little more complicated by the use of a docker image. My hunch is I probably need to install the drivers and toolchain outside Docker, which took me to the AWS NVIDIA driver install guide. I’m not sure how that matches up with what CmdStan expects. In any case, when I figure out the full toolchain install I’ll post an update for posterity.

3 Likes

hi @nerutenbeck , I’m going to add a stan blog post to set up rstudio and stan on ec2. Do you have any interest in doing one to set up gpu?

3 Likes

@spinkney I’d love to! Thanks for the offer. DM me to briefly discuss timing and details? I’m also on the Stan slack channel if you’d rather discuss there.

Did the stan on ec2-gpu blog post ever go up? If so could you provide a link?