Calling mac M1 users to try pip-installable httpstan branch

@ariddell and I discussed making httpstan pip installable from source. This should allow M1 users (and more generally any architecture for which pre-compiled wheels do not exist) to use httpstan (which is required for the most recent version of pystan).

There is a branch which should achieve this goal, and it would be interesting to see whether it works for others or just for me. If you have an M1 mac (or another architecture), could you please run the following from the console and report back as to whether it works?

# Uninstall httpstan in case it's already installed.
pip uninstall -y httpstan pystan
# Install httpstan from source (this will take a while).
pip install -v git+https://github.com/tillahoffmann/httpstan@pip
# Install pystan.
pip install 'pystan>=3'
# Try a dummy program (this should report build and sampling progress).
python -c 'import stan; stan.build("parameters { real x; } model { x ~ normal(0, 1); }").sample()'
7 Likes

Hi just to comment that this worked for me, I used it in a new virtual env

3 Likes

Hi! Thanks for this, I am trying to get PyStan working on my M1 Macbook and came across your discussion here

First I create a new conda environment and install pip

conda create -n new --yes
conda activate new
conda install pip --yes

which python3                               # run to check it's the correct pip
python3 -m pip install -U pip   # update pip

I get the following error when running your code from the line:

pip install -v git+https://github.com/tillahoffmann/httpstan@pip
Using pip 22.0.3 from /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages/pip (python 3.9)
Collecting git+https://github.com/tillahoffmann/httpstan@pip
  Cloning https://github.com/tillahoffmann/httpstan (to revision pip) to /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d
  Running command git version
  git version 2.30.0
  Running command git clone --filter=blob:none https://github.com/tillahoffmann/httpstan /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d
  Cloning into '/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d'...
  Running command git show-ref pip
  0ba90d4be04967d1a9200a8ccb4be11fab788abd refs/remotes/origin/pip
  Running command git symbolic-ref -q HEAD
  refs/heads/main
  Running command git checkout -b pip --track origin/pip
  Switched to a new branch 'pip'
  Branch 'pip' set up to track remote branch 'pip' from 'origin'.
  Resolved https://github.com/tillahoffmann/httpstan to commit 0ba90d4be04967d1a9200a8ccb4be11fab788abd
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Downloading setuptools-60.9.2-py3-none-any.whl (1.1 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 7.8 MB/s eta 0:00:00
  Collecting poetry_core>=1.0.0
    Downloading poetry_core-1.0.7-py2.py3-none-any.whl (424 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 424.8/424.8 KB 11.7 MB/s eta 0:00:00
  Installing collected packages: setuptools, poetry_core
  Successfully installed poetry_core-1.0.7 setuptools-60.9.2
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: aiohttp<4.0,>=3.7 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from httpstan==4.6.1) (3.8.1)
Requirement already satisfied: marshmallow<4.0,>=3.10 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from httpstan==4.6.1) (3.14.1)
Requirement already satisfied: setuptools>=41.0 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from httpstan==4.6.1) (58.0.4)
Requirement already satisfied: webargs<9.0,>=8.0 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from httpstan==4.6.1) (8.1.0)
Requirement already satisfied: appdirs<2.0,>=1.4 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from httpstan==4.6.1) (1.4.4)
Requirement already satisfied: numpy<2.0,>=1.16 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from httpstan==4.6.1) (1.22.2)
Requirement already satisfied: attrs>=17.3.0 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from aiohttp<4.0,>=3.7->httpstan==4.6.1) (21.4.0)
Requirement already satisfied: frozenlist>=1.1.1 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from aiohttp<4.0,>=3.7->httpstan==4.6.1) (1.3.0)
Requirement already satisfied: aiosignal>=1.1.2 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from aiohttp<4.0,>=3.7->httpstan==4.6.1) (1.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from aiohttp<4.0,>=3.7->httpstan==4.6.1) (6.0.2)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from aiohttp<4.0,>=3.7->httpstan==4.6.1) (2.0.12)
Requirement already satisfied: yarl<2.0,>=1.0 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from aiohttp<4.0,>=3.7->httpstan==4.6.1) (1.7.2)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from aiohttp<4.0,>=3.7->httpstan==4.6.1) (4.0.2)
Requirement already satisfied: packaging in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from webargs<9.0,>=8.0->httpstan==4.6.1) (21.3)
Requirement already satisfied: idna>=2.0 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from yarl<2.0,>=1.0->aiohttp<4.0,>=3.7->httpstan==4.6.1) (3.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages (from packaging->webargs<9.0,>=8.0->httpstan==4.6.1) (3.0.7)
Building wheels for collected packages: httpstan
  Running command git rev-parse HEAD
  0ba90d4be04967d1a9200a8ccb4be11fab788abd
  Running command Building wheel for httpstan (pyproject.toml)
  running build
  running build_py
  creating /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build
  creating /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9
  creating /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/build_ext.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/config.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/fits.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/models.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/openapi.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/cache.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/__init__.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/schemas.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/utils.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/services_stub.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/compile.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/app.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/__main__.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/views.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/routes.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  creating /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan/services
  copying httpstan/services/arguments.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan/services
  copying httpstan/services/__init__.py -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan/services
  copying httpstan/socket_writer.hpp -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/socket_logger.hpp -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/stan_services.cpp -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/empty.cpp -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/stan.pyi -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan
  copying httpstan/services/cmdstan-help-all.json -> /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build/lib.macosx-10.9-x86_64-3.9/httpstan/services
  running build_ext
  downloading build/archives/math-v4.2.1.tar.gz
  extracting archive build/archives/math-v4.2.1.tar.gz
  tar -C build -zxf build/archives/math-v4.2.1.tar.gz
  touch build/math-4.2.1
  /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile.libraries build/math-4.2.1/lib/sundials_5.7.0/lib/libsundials_nvecserial.a
  /bin/sh: x86_64-apple-darwin13.4.0-clang++: command not found
  x86_64-apple-darwin13.4.0-clang++ -fPIC -pipe -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT  -O3 -I build/math-4.2.1/lib/sundials_5.7.0/include -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include build/math-4.2.1/lib/sundials_5.7.0/include/stan_sundials_printf_override.hpp build/math-4.2.1/lib/sundials_5.7.0/src/nvector/serial/nvector_serial.c -o build/math-4.2.1/lib/sundials_5.7.0/src/nvector/serial/nvector_serial.o
  make[1]: x86_64-apple-darwin13.4.0-clang++: No such file or directory
  make[1]: *** [build/math-4.2.1/lib/sundials_5.7.0/src/nvector/serial/nvector_serial.o] Error 1
  make: *** [build/math-4.2.1/lib/sundials_5.7.0/lib/libsundials_nvecserial.a] Error 2
  Traceback (most recent call last):
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/setup.py", line 36, in <module>
      setup(**setup_kwargs)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 155, in setup
      return distutils.core.setup(**attrs)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build.py", line 26, in run
      subprocess.check_call(["make"], cwd=self.build_lib)
    File "/Users/tommylees/miniconda3/envs/test/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['make']' returned non-zero exit status 2.
  Traceback (most recent call last):
    File "/Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/poetry/core/masonry/api.py", line 68, in build_wheel
      return unicode(WheelBuilder.make_in(poetry, Path(wheel_directory)))
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py", line 72, in make_in
      wb.build()
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py", line 103, in build
      self._build(zip_file)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py", line 135, in _build
      self._run_build_command(setup)
    File "/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-build-env-lvr_ot6t/overlay/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py", line 163, in _run_build_command
      subprocess.check_call(
    File "/Users/tommylees/miniconda3/envs/test/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/Users/tommylees/miniconda3/envs/test/bin/python3', '/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/setup.py', 'build', '-b', '/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d/build']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Building wheel for httpstan (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/tommylees/miniconda3/envs/test/bin/python3 /Users/tommylees/miniconda3/envs/test/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/tmp4r3qtbuf
  cwd: /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-gipaio1d
  Building wheel for httpstan (pyproject.toml) ... error
  ERROR: Failed building wheel for httpstan
Failed to build httpstan
ERROR: Could not build wheels for httpstan, which is required to install pyproject.toml-based projects

The important bits:

  subprocess.CalledProcessError: Command '['/Users/tommylees/miniconda3/envs/new/bin/python3', '/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-cpw0z7y5/setup.py', 'build', '-b', '/private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-cpw0z7y5/build']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Building wheel for httpstan (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/tommylees/miniconda3/envs/new/bin/python3 /Users/tommylees/miniconda3/envs/new/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/tmpby8qbgce
  cwd: /private/var/folders/q3/0lmt64ld10s14_0n0vxpt_m00000gp/T/pip-req-build-cpw0z7y5
  Building wheel for httpstan (pyproject.toml) ... error
  ERROR: Failed building wheel for httpstan
Failed to build httpstan
ERROR: Could not build wheels for httpstan, which is required to install pyproject.toml-based projects

Any advice?

Seems to be fixed by restarting my terminal! Thanks so much !