Stan in R on PowerPC G5 (10.6 PPC): Help needed

Developers
21

I would be okay with 64 bit (PowerPC G5), but Linux or rather BSD is a step I can take if everything fails on MacOS.
For one thing, it will be inconvenient for practical reasons (rebooting into other OS just for the sake of running R). From another side, making it work on MacOS will probably benefit someone else who faces a similar issue. It is sad that many developers give up on PowerPC, since G5 ones are pretty capable even nowadays.

(I have a backup route with Julia instead of R, but perhaps all the same issues with Stan will arise in that case too.)

22

Yes, Julia will have all the same issues as it just calls cmdstan.

The errors above are due to TBB, both because it’s looking for clang and because it’s assuming that you’re using 64bit powerpc. You can use Stan to force the choice of c compiler with the TBB by creating a make/local file and adding: TBB_CC = /opt/local/bin/gcc-mp-7

However there’s no easy resolution for the error:

/usr/include/mach/task.h:35:10: fatal error: mach/ppc64/task.h: No such file or directory
 #include <mach/ppc64/task.h>
          ^~~~~~~~~~~~~~~~~~~

As that is caused by some part of the TBB’s build/config process that Stan doesn’t interact with.

23

I have some success in proceeding toward building rstan, though V8 failed at the moment: Override dependency on V8 or make V8 build on PowerPC Mac? (R 4.1.2)

If that is sorted and I am ready to build rstan itself, all the same issues as with CmdStan will arise, right? I am thinking how to minimize “dependency hell”, since fixing every failing component is usually a non-trivial task in its own right.

24

If you have an option to work with a more recent CPU & operating system than powerpc G5 & Mac OS X 10.6 , I suspect things will become much easier. It looks like many open source tools and libraries no longer maintain support for that platform, so it might be simpler to switch to a different platform that is supported.

25

@andrjohns I have made through building all required libraries with Macports:

36-9:~ svacchanda$ port installed tbb
The following ports are currently installed:
  tbb @2020.3_0 (active)
36-9:~ svacchanda$ port installed eigen
The following ports are currently installed:
  eigen @2.0.17_0 (active)
36-9:~ svacchanda$ port installed boost
The following ports are currently installed:
  boost @1.76_0 (active)
36-9:~ svacchanda$ port installed sundials
The following ports are currently installed:
  sundials @5.7.0_0+accelerate+gcc7+mpich (active)

How do I use these for CmdStan in place of its own attached libs?

26

There are make variables you should be able to set to point cmdstan to them, but they’re sadly not that documented anywhere. @rok_cesnovar has been able to help me with some in the past, I know for TBB you need to set TBB_INC and TBB_LIB to point to the path with the headers and library respectively, and I suspect there are the same for other libraries

27

It looks like the macports Eigen is very out of date (we’re currently on 3.3.9), and won’t be compatible with Stan. If the TBB is the only library you can’t build locally, then you should use the TBB_INC and TBB_LIB flags that @WardBrian mentioned. There’s some more guidance on those in this section of the Github readme.

Unfortunately we don’t have equivalent functionality for the other libraries, only the TBB

1 Like
28

I have finally built gcc10 and gcc11 natively on PowerPC, however CmdStan still fails with the same lgamma error when I tried to build using gcc10. This looks like a problem on CmdStan side.

29

For the Boost lgamma fix to work, you need to use the github version of cmdstan (the changes aren’t in a release yet).

Then, in your make/local file, add:

CXXFLAGS += -D_BOOST_LGAMMA
1 Like
30

Thank you! I will try and update you.

I wanna rebuild R using gcc11 outside of Macports also. At the moment there is some issue with building 64-bit on 10.5.8, but 32-bit works both on 10.5.8 and 10.6 PPC.

31

Wow! According to Wikipedia, those are from 2003 to 2006! It’s amazing anything still runs on those, because Apple’s not exactly Microsoft in terms of its commitment to backwards compatibility.

32

I have finally fixed TBB, mostly. Some extra work needed, but it builds and passes 78% of the test suite. Failures are likely a result of incorrect alignments – the source code does not have PPC in mind.
Eigen builds fine, and we have the latest version: Install eigen3 on macOS with MacPorts

v8 is a no-go for now. This is the only dependency which is unlikely to be fixed any soon, if ever. (I would like to fix Node for PPC, but it is a time consuming stuff, and not the number 1 priority due to uncertainty of success.)

33

Note that the TBB issues are unlikely to be caused by the PPC architecture alone, since it passes all tests on Linux PPC

34

ABI matters, not just the physical CPU. And ABI is very different from ELF. In particular, and what is relevant (since we don’t have assembler in TBB anymore), are alignments and paddings. And those differ from ELF.

35

I find this thread to be amazing. Maybe we should have a competition for the oldest computer that can install and run Stan?

36

Late G5 are reasonably fast and perfectly usable for many tasks to this day (aside of fancy web apps). There is no reason not to have Stan on PowerPC, when we have the latest compilers, Cmake, mpich, R, openblas and most of math software. I have recently fixed OpenCoarrays! :)

37

@WardBrian @andrjohns I have been reading docs and writing portfiles code for the past 20 hours, and I would appreciate some clarification.

If I understand correctly, the current structure of CmdStan requires:

  1. stanmath (header-only lib)
  2. stan (header-only lib) – why these two are separate by the way?
  3. stanc3 (Stan compiler written in OCaml)
  4. cmdstan (interface built on top of the three).

stanmath has a runtime dependency on eigen3 and tbb, stan depends on stanmath and rapidjson, stanc3 has a bunch of OCaml dependencies, cmdstan depends on all of these plus sundials, cli11 and gmake to build models.

Ideally, we (in Macports) do not want to have duplicate libraries, unless absolutely necessary, so it is desirable to have these as separate ports with correct dependencies specified (i.e. build, lib, run and test dependencies).
Should I edit build targets in order to use external (in Macports prefix) dependencies?
If there is some undocumented stuff like you mentioned earlier, that would be really helpful to know.

Aside of correct structure of dependencies, this part looks problematic: “Make only works properly when invoked from the <cmdstan-home> directory; attempts to use this Makefile from another directory by specifying the full path to the file makefile won’t work”.

This seems to imply that it needs makefiles at the runtime and the executable is run from within the same environment where it was built.
In Macports the build directory is removed, once the port is installed. What we can do is create a cmdstanhome directory, say, in /opt/local/libexec, copy the tree structure there (folders and makefiles) and symlink needed includes, libs and stanc3 executable from their respective locations. Does not look like a neat solution, but if that can work, not too bad.

38

@andrjohns is working on Makefile refactors which will allow you to specify all those libraries for Math. It should be fairly easy to do the same for rapidjson and cli11 if required, but those are header only so it might matter less.

Your observation is correct, using CmdStan requires the end user have a C++14 compiler and make. The solution you describe sounds reasonable, and is similar to what e.g the conda distribution uses.

My guess is the hardest part may be the OCaml dependencies. I’m not sure what the status of Mac PowerPC OCaml distributions is, but getting all of our dependencies to cross compile for Windows was nontrivial

39

Thank you for responding. I have fixed OCaml bytecode compiler for PPC not long ago, and it has been merged in Macports. That is, we need to use ocamlc and not ocamlopt on PPC. Dependencies normally build; it may require to specify correct target though, since by default some ports try to build two targets, which is not possible with ocamlopt being unavailable (usually fix is trivial and amounts to an extra line in a portfile).

Native compiler might be fixable with OCaml 4.x.x, though it is likely not to be very easy. I have done that half way, but assembler should be worked on somewhat further. I have been told though that for OCaml 5 we may be out of luck with PPC for ocamlopt. If so, then ocamlc is what we have.

40

That would be awesome! And gonna make the work on our end much easier.

Having compilers and other tools is not an issue; all GCC versions build fine at least on 10.5.8 and 10.6 PPC, and GCC upstream supports Darwin PPC (this does not extend to extra langs like Go and Rust, but C, C++ and Fortran are good; Fortran needs some improvement to IEEE support, which I am currently working on).
(Thankfully, Stan developers support GCC – Clang is broken on PPC forever, though libcxx is likely fixable.)
My concern was with setting up user environment that would comply with CmdStan documentation and fit Macports routines, but this I can handle on my end.

I will update re OCaml packages status soon. Most of needed stuff did build. I would expect the rest also builds fine. As for stanc3, we need to try and see :)