Temporary folders not being cleaned up

Interfaces RStan
1

Short summary of the problem
Over the past couple of years, I have been developing an R package that has several functions that call pre-compiled Stan models (using the rstan sampling function). Recently I have been using some of these functions in a simulation study with computations being done on several different Windows computers. I soon realized that large numbers of randomly named folders (with names like RtmpIB8N2A) were being left behind in the AppData\Local\Temp, even when everything terminated correctly and R was exited normally. The folders contain one or two dll files.
Is this expected behavior? Is there an easy way to force cleanup while the simulation is running?
I am using R version 3.6.2 and rstan version 2.19.2.

code_to_run_your_model(if_applicable)

If possible, add also code to simulate data or attach a (subset of) the dataset you work with.

Don’t forget to add relevant tags to your topic (top right of this form) especially for application area. Delete this text before posting your question :-) Thx!

If your question relates to installation please provide the following information:

  • Operating System
  • RStan Version
  • Output of writeLines(readLines(file.path(Sys.getenv("HOME"), ".R/Makevars")))
  • Output of devtools::session_info("rstan")

If you are reporting a bug (thank you!) please use the issue tracker (github.com/stan-dev/rstan/issues) instead of the forums.

Looking forward to your topic!

2

Hmm, I’m not sure. I’ll probably have to ping someone else, but first a couple Qs – what are the names of the files left behind? And does rstan 2.21 do the same thing?

Are you developing your package through rstan or rstantools (GitHub - stan-dev/rstantools: Tools for Developing R Packages Interfacing with Stan)?

1 Like
3

I tried again using R 3.6.3 and rstan 2.21.2 and got the same thing. I just found out about rstantools last week but have not used it yet—motivated to do so o learn how to compile my Stan models on install.

I went to a simpler example (still complicated) generating draws for just one data/model combination. I am using options(mc.cores = parallel::detectCores())
and running four chains simultaneously. When it starts running four new folders show up in Temp. After completion, it only one of those is deleted. The names and contents of the remaining three folders is given below.

RtmpGcDSc3
file417012f25d17.dll

Rtmpi2THys
file4170567ff8.dll
file417012f25d17.dll

RtmpY3K55m
file4e5c70642f9.dll

4

Sorry for the delay!

@andrjohns Do you know where these temporary files come from and if there’s an easy way to clean them up or not?

Edit: I guess also does it makes sense that rstan would be making them?

5

It seems almost certain that they are being generated by something in rstan. The way I delete them is to go to the folder and do CTL-A, Delete!

6

One more thing. I have two co-authors running the same simulations on Linux and they say they see no evidence of such files (but I wonder if they are looking in the right places).

7

Atleast on Python side and the temporary files have some issues. The default way of working with them causes them to stay “open” until python process stopped. This means one can not delete temp files from python process on Windows.

(There are ways to handle these, but these things can be tricky)

1 Like
8

just confirming that I’ve seen this behaviour for years… have to remember to delete the temp folder every so often.

1 Like
9

It makes sense to make this an rstan issue: Issues · stan-dev/rstan · GitHub . Maybe someone watching the issues will have a better idea if there is a more automated solution that clearing temp.

(Edit: or better yet come up with a fix :P)

10

Just a final comment on this. Working with one of my co-authors we have determined that rstan does proper cleaning of such directories from /tmp in Linux—so it is a Windows problem. This behavior might not be a problem for most, but my simulation was using five calls to sampling in each simulation iteration and with running four chains at a time, it was building up a lot of left-over folders! I learned about the problem when the three computers I was testing/running on all told me that free space on my drives was seriously low. I wrote some R functions to clean up myself after each simulation iteration, so now I am OK again.

1 Like
11

Any chance you’d be willing to share the code you used to clean up your temporary files? I’ve encountered the same issue, and I just had to delete ~20GB of temporary R folders (somewhere in the ballpark of 42000+ folders) that weren’t deleted from my simulation study over the last couple of days.

Curiously, I’ve never had this issue in brms, which I tend to use much more frequently than rstan. I can’t imagine why it wouldn’t be an issue for brms but would be for rstan since in both cases the compiler is called and sampling is done

12

Sure. Here is what I use. It is a little messy because I only commented out some of the debugging stuff that I used.

If you are running a simulation using these cleanup tools, you may have problems starting up another R instance on the same computer because its Rtmp file will be deleted.

Bill

I call the following just before the simulation loop to make sure I do not delete pre-existing files.

13

Looks like my email got truncated. Here it is again:

I call the following just before the simulation loop to make sure I do not delete pre-existing files.

finding the pre-existing Rtmp files

if(R.version$os==“mingw32”)existing.folders ← findTempFolders()

Then, I call

clean up new Rtmp folders before starting the next iteration

if(R.version$os==“mingw32”){

if(itrial>1)deleteTempFolders(existing.folders)

}

The functions that do the work are:

findTempFolders ← function(){

Function to find the names of all current Rtmp folders

###Get a sample Rtmp path/name and extract the path

tmpdir ← tempdir()

num.char.in.name ← nchar(tmpdir)

the.path ← substring(tmpdir, 1, num.char.in.name-10)

get a list of all of the Rtmp folders looking like RtmpktljJ4

###all.names2 ← system2(paste("ls ", the.path), stdout = TRUE)

the.command ← paste("ls ", the.path)

###cat(the.command, “\n”)

all.names ← system(the.command, intern=TRUE)

name.lengths ← nchar(all.names)

assumes six random letters/numbers in the names

the.ones ← substring(all.names, 1, name.lengths-6)==“Rtmp”

folder.names ← all.names[the.ones]

folder.names

}

deleteTempFolders ← function(old.folders){

Function to delete all of the new Rtmp folders

###Get a sample Rtmp path/name and extract the path

tmpdir ← tempdir()

num.char.in.name ← nchar(tmpdir)

the.path ← substring(tmpdir, 1, num.char.in.name-10)

###Get a sample Rtmp path/name and extract the path

current.folders ← findTempFolders()

which.new ← !is.element(current.folders, old.folders)

new.folders ← current.folders[which.new]

get a list of all of the Rtmp folders looking like RtmpktljJ4

the.command ← paste(“rm -r “, paste(the.path, new.folders, sep=””, collapse=" "))

###cat(the.command, “\n”)

###system2(the.command)

###browser()

protect in case there are no new folders

if(length(new.folders)>0)system(the.command, intern=TRUE)

invisible()

}