Rstan based package and winbuilder -- out of memory allocating

Developers
21

This turned out to be harder than expected to find a solution that I could make work, but using two stan_files directories (one for 32 bit) is adequate.

ifeq "$(WIN)" "64" 
SOURCES = $(wildcard stan_files/*.stan)
else
SOURCES = $(wildcard stan_files32/*.stan)
endif
22

Glad something is working. Did consolidating declarations in generated quantities not work?

23

I did some but apparently not enough ā€“ the file is already obscene, I didnā€™t really want to turn a lot of meaningful names that other functions may depend on into indices.

24

This solution fails to take into account 32 bit Solaris, and CRAN is unhappyā€¦ Iā€™m at a loss how to modify the makevars to cope with this, any ideas would be great.

25

Is it necessary to use the stan_files32 directory outside of Windows?


My hope is that g++-5.x on 32 bit solaris is efficient enough to compile what is in stan_files even though g++-4.9 on 32 bit Windows is not. You can try on
https://builder.r-hub.io/advanced

27

Regarding ā€œout of memoryā€ errors: does anyone know how strict CRAN is with this error when submitting a new package version? The blavaan package is currently passing checks locally, on travis, and on win-builder. But the CRAN checks currently show an ā€œout of memoryā€ windows error:

https://www.r-project.org/nosvn/R.check/r-release-windows-ix86+x86_64/blavaan-00install.html

Iā€™d like to know whether I can ignore it.

28

I donā€™t ā€˜knowā€™, but my sense with this sort of thing is that you might slip by for a while and then one day youā€™ll get flagged for full inspection, dragged out the back and beaten up, and have to fix it and a bunch of other obscure stuff you had no idea about ;) I wish win32 could just be treated as itā€™s own separate OS like Solaris or somethingā€¦ would be nice to be able to include the vignette etc that depends on stan.

1 Like
29

My recent experience is that even for NOTES that happen only on one platform (say for examples taking 11s instead of < 10s), the package gets automatically rejected. So I assume that for an out of memory error the same would happen. I donā€™t know what would happen if you ā€œappealedā€ by writing to the R-devel mailing list.

30

I canā€™t imagine CRAN letting that slide. But it is probably fixable by making the Stan program a bit smaller and / or possibly utilizing standalone generated quantities.

31

I havenā€™t confirmed it but there may be a bug in rstan::gqs() for some kinds of parameter types: https://github.com/stan-dev/rstan/issues/695. Even if so, that could still be a potential long term solution when fixed.

Edit: Iā€™ve checked and I do think thereā€™s a bug. rstan::gqs() is only currently working for scalars. So I donā€™t think doing standalone generated quantities is a short term option.

32

Thanks, the gqs() function for vector/matrix parameters should indeed be useful for the future.

33

You can probably use the thin version of standalone generated quantities currently, although it is probably easier to just consolidate the Stan program a little bit so that 32bit R can compile it without using more than 2GB of RAM.

34

Itā€™s fixed on GitHub now if you want to try it out, but wonā€™t help with CRAN until the next release of rstan.

35

Maybe I am missing something obvious, but what does ā€œthin versionā€ refer to? I see a ā€œgqs_lightā€ function in rstan, but it is not exported so could not be used to make CRAN happy. But maybe you mean something elseā€¦

36

Thatā€™s the one. It is somewhat experimental, but it is legal to use rstan::: in build / install scripts.

37

I have gotten back to trying out gqs and gqs_light on the big blavaan Stan file and am encountering some errors. I am attaching some files to reproduce the errorsā€¦ see gqs_code.R. I donā€™t expect anybody to debug the giant Stan files, but I am wondering whether I am making an obvious mistake in calling gqs, or possibly the problems arise when a transformed parameters block is in the Stan file, or ā€¦

I can confirm that the Stan file works when it is all a single file (when the gqs block is in the main file). Thanks for any advice.

gqs_code.R (1.1 KB) semstan.r (3.2 KB) stanmarg_gqs.stan (28.3 KB) stanmarg_nogq.stan (24.9 KB)

38

Can you post the error message?

39

The error messages are pasted in comments in the R file:

From gqs(), I am getting

Error in draws[, draws_colnames, drop = FALSE] : subscript out of bounds

From gqs_light(), I am getting

Error in if (!after) c(values, x) else if (after >= lengx) c(x, values) else c(x[1L:after], : argument is of length zero

which I believe is coming from a call to append()

40

OK. I could understand if the unexposed gqs_light got broken by the switch to CRTP but gqs should be working. If you set

options(error = recover)

before calling gqs, you can get into the right function call and see what element of draws_colnames is not among colnames(draws). But it is probably some bug.

41

Thanks, that helps. It looks like draws_colnames contains both parameter names and transformed parameter names. So, when sampling from the model, the pars argument should include parameters and transformed parameters. But if I do that, I get an error from src/stan/services/sample/standalone_gqs.hpp (ā€œWrong number of parameter values in draws from fitted modelā€, around line 92). There, I think it is expecting only parameter draws, without transformed parameters.