Using Make in PowerShell to build CmdStan on Windows 10

General
,
1

Hey all,

So I’m working on creating an msi installer using NSIS for Cmdstan.
One of the things I’m trying to do is to make the process happen within the Powershell terminal.
One of the nice things about the Windows 10 Powershell is that curl and tar come with the powershell
so I’ve been using them to help grab cmdstan from the latest release page and untar it into a directory.
The next thing I need to do is use make to build out cmdstan and
the command I’m using is

make -j2 build

The result I get in the Powershell terminal is

stan/lib/stan_math/make/libraries:83: *** Recursive variable `findfiles’ references itself (eventually). Stop.

I’m reaching to see if there’s something I’m forgetting to do or if there’s a specific version of Make I’m supposed to use.

Also, I’m trying to not use Rtools to do this.

Thanks again and any feedback is much appreciated and welcome.

2

Also I’m in the cmdstan directory when I use make.

3

the call findfiles is something we define here:

if you look at the blame you’ll see that we used to distinguish between Windows_NT and other. it’s possible that the generalized version didn’t anticipate running in the powershell? @rok_cesnovar ?

1 Like
4

There is no issue with powershell, I have actually almost never used the regular command prompt on windows. Cmdstanr tests use powershell without issue as well.

Git blame is weird in this case, the last three commits are all part of the same pull request (Cleanup the use of find in makefile by rok-cesnovar · Pull Request #1567 · stan-dev/math · GitHub), we went through a couple of versions before settling on this one. Before that, cmdstan relied on the GNU tool find, which does not come with Windows by default.

Another thing is you should be using mingw32-make on Windows, but that is not the culprit here.

This seems to be some make that does not support recursive calls. So its definitely not the Rtools supplied one.

EDIT: Oh, I see you are trying to not use RTools. It seems the make you installed does not support recursive function calls in makefile, which is a deal breaker for cmdstan at the moment.

We obviously accept PRs to replace findfiles with anything else, but it has to work for both command prompt and powershell and be available without additional dependencies.

2 Likes
5

Hey @mitzimorris and @rok_cesnovar, thanks again for the feedback I appreciate it. So I didn’t realize that there was a change in the makefile in regards to findfiles and gnufind. The version of make I was using was very old it looks like but it was usable in the PowerShell.
Here’s the link to what I was using.
http://gnuwin32.sourceforge.net/packages/make.htm

So it looks like a really old version that still seems to allow you to use make in the PowerShell terminal.
I’ll take a look to see if there’s a way around this maybe by looking at how to copy the mingw32-make exe file and run it in either the command prompt or PowerShell.
Something like that was done here.

I’ll try and also test with gnufind as well just in case it works.

Thanks again guys, much appreciated!

6

Also, @rok_cesnovar can you post the link to where exactly the tests are being done with PowerShell for CmdstanR, it might help as well. Thanks again.

7

This isnt really a recent change, I think at least pre 2.23 which is where I took a hammer to
all the various uses of GNU tools in our makefiles that caused issues on Windows.

Prior to that cmdstan required a bunch of GNU tools to run.

This entire GHA script uses powershell: https://github.com/stan-dev/cmdstanr/blob/master/.github/workflows/R-CMD-check.yaml

As do Jenkins tests here: https://jenkins.mc-stan.org/blue/organizations/jenkins/CmdStan/detail/develop/913/pipeline

But both of these tests rely on having RTools installed so not sure its helpful.

@bparbhu If Rtools is not an option I think we should be relying on barebones Msys2. Or do you have other plans or directions you are exploring?

1 Like
8

Thanks @rok_cesnovar, much appreciated! So the general direction I would like to go if possible is to not use MSYS2 or Rtools. My first idea was to figure out how to launch MSYS2 commands from NSIS scripts and I think I figured it out but ideally it would be better to have an executable generated that people can click and install without having to rely on those two things needed as a prerequisite. Though, if this route fails then the next idea would be to use MSYS2 like this through the NSIS script I’m doing through Execshell commands in NSIS.

1 Like
9

Cool.

I feel Stan is a bit too complex of a package and uses some complex libraries (sundials and TBB in particular) that will make it hard to work without tools like msys2, cygwin or something along those lines.

If we can make it work, that would be fantastic though.

2 Likes
10

agreed - it’s not that we don’t want to use MSYS2 - we need to use the minimal subset of MYS2 tools needed to make GNU-make and makefiles fly. this includes the correct C++ compiler and whatever unix utilities are used by the makefiles.

1 Like
11

So I tried switching over to the chocolatey package manager for PowerShell. It acts like an apt-get for windows PowerShell and command prompt. One of the nice things about it is that it includes an updated gnu make command that is version 4.3. So the first thing I did was type the command

make clean-all

and the the result seems to work but for some reason there was a file pattern it couldn’t detect.

The next command I’m doing is

make -j2 build

and I get the error

image
image3166×1324 271 KB

So still trying to figure this out but I’m not getting the same error I was before from the older version of make for windows I was using before.

The error looks like it might be in regards to child processes not finishing but I’m not totally sure about what I’m seeing.

Again any feedback is very welcome and I appreciate all the help.

Thanks again!

12

maybe this could be another findfiles error?

13

I think we should strive for an installer which provides a known, trusted C++ compiler, ditto known versions GNU-make, and whatever Unix utilities are needed by the Makefile rules - i.e., as much control over the toolchain as possible.
leaning on Powershell or Chocolatey puts us in the Mac OS bind, where updates to Xcode and the OS mess with our installer scripts.
that said, a working prototype would be nice just to let folks play around with the experience.

1 Like
14

True I agree with you @mitzimorris , the reason I went on this detour is that I was curious about just testing this out as I couldn’t find any updated make command that was maintained after 2006 that could work in PowerShell. I’m going back to just trying to do this through MSYS2 at this point. Though, If I find a way to make this work in PowerShell later I’ll definitely release it to people.

1 Like
15

Hmmm…something I might look into for the future is whether or not there are issues with file paths and whether or not there are expectations around “\” versus “/” for windows 10 in the PowerShell.