Pystan 3, gcc compilers and Docker images

Interfaces PyStan
1

Just thought I’d give a post on some issues that I had setting up PyStan 3, and how I got around them, in case there’s someone else this might help.

I tried to get PyStan 3 set up to run in a Docker container, but was getting C++ compiler error messages about an undefined symbol nested in a lengthy Python error traceback when executing the stan.build() method. The reason for this seems to be a dependency on gcc ≥9.0 (or clang ≥10.0) - this is now stated in the documentation to the httpstan module, a dependency of PyStan 3. The python:latest Docker Hub image (at time of writing) uses Debian Buster with gcc version 8.3.0; updating this to the latest gcc resolved the issue for me. I’ve included an example of the error messages that I was receiving with the out-of-date gcc compiler, and an example Dockerfile that sets up a working PyStan 3 container.

I was doing this specifically within a Docker container, but I suspect the error will arise for anyone using an out-of-date compiler. The only change I needed was to update to a newer gcc version, and the PyPI wheels worked fine after that.

Example error message

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.9/site-packages/httpstan/views.py", line 253, in handle_show_params
    services_module = httpstan.models.import_services_extension_module(model_name)
  File "/usr/local/lib/python3.9/site-packages/httpstan/models.py", line 90, in import_services_extension_module
    module: ModuleType = importlib.util.module_from_spec(spec)  # type: ignore
  File "<frozen importlib._bootstrap>", line 565, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1108, in create_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: /root/.cache/httpstan/4.4.0/models/ekul4lrl/stan_services_model_ekul4lrl.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNSt19basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/stan/model.py", line 426, in build
    return asyncio.run(go())
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/stan/model.py", line 415, in go
    raise RuntimeError(resp.json()["message"])
  File "/usr/local/lib/python3.9/site-packages/stan/common.py", line 24, in json
    return simdjson.loads(self.content)
  File "/usr/local/lib/python3.9/site-packages/simdjson/__init__.py", line 61, in loads
    return parser.parse(s, True)
ValueError: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc.

Example Dockerfile for a functional PyStan 3 container

FROM python:latest

# Need to install the updated gcc compiler for Pystan 3
# Add the Debian source list
RUN echo 'deb http://deb.debian.org/debian testing main' >> /etc/apt/sources.list

# Update the repos
RUN apt-get update -y

# Install the latest gcc (should be version 10)
RUN apt-get install -y gcc

# No need to install further packages so remove the lists to save space
RUN rm -rf /var/lib/apt/lists/*

# Install PyStan 
RUN pip install --pre pystan

Hope this helps anyone else who may have a similar issue.

4 Likes
2

If anyone is trying to get the above Dockerfile to work now-ish (2021 / 10 / 16 ) , it seems to fail as one or more of pystan / httpstan have not published packages for python 3.10 to pypi , so pip searches for very old versions of pystan and httpstan ( maybe before the published packages were tagged indicating specific python version requirements ?), then tries to build them from source, and fails for some reasons.

Here’s a variation of the above Dockerfile that works for me, by pinning which python we use in our base image to the older python 3.9.7:

FROM python:3.9.7-bullseye

# Need to install the updated gcc compiler (version 10) for Pystan 3
RUN echo 'deb http://deb.debian.org/debian testing main' >> /etc/apt/sources.list
RUN apt-get update -y
RUN apt-get install -y gcc
RUN rm -rf /var/lib/apt/lists/*

RUN python3 -m pip install httpstan==4.6.0
RUN python3 -m pip install pystan==3.3.0

edit: here’s a rough demo workflow to build and run a stan model using pystan inside a container, where scripts and other files can be kept in the host machine filesystem, and we run pystan inside an ephemeral container which has is given access to those scripts

1 Like
3

Python 3.10 wheels have not been released yet. Waiting on a macOS bug to be fixed (BUG: Missing 3.10 wheels for macOS · Issue #20045 · numpy/numpy · GitHub). (The wheel release process makes macOS and Linux wheels together – currently it’s a little complicated to publish linux wheels separately.)

4

Python 3.10 wheels are available now.

2 Likes
5

That’s great, thanks! I think (at time of writing) the python:3.9 and python:latest (i.e. python:3.10.0-bullseye) Docker images are now running on Debian Bullseye, which has gcc 10, so it’s no longer necessary to update gcc and it should be sufficient to just pip install pystan directly in the new Python Docker images.

FROM python:3.10.0-bullseye

# Install PyStan 
RUN pip install pystan
3 Likes