# Specifying output file names

**URL:** <https://discourse.mc-stan.org/t/specifying-output-file-names/16345>\
**Category:** CmdStan\
**Tags:** cmdstanpy\
**Created:** [July 2, 2020, 12:39pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345 "2020-07-02T12:39:25Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 12:39pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/1 "2020-07-02T12:39:25Z")

</div>

I am working on a distributed computing pool and wish to run many independent instances of an analysis. If these jobs are submitted at the same time, I will get file-name clashes and some results will be overwritten. Is there someway to specify the name of the output CSV file in CmdStanPy when it is copied back over from /tmp directory?

- Operating System: Ubuntu 18.04.4 LTS
- CmdStan Version: 2.23.0
- Compiler/Toolkit: g++

---

<div class="post-metadata">

**Author:** ![jonah](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jonah/32/200_2.png) [@jonah](https://discourse.mc-stan.org/u/jonah)\
**Post date:** [July 2, 2020, 2:53pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/2 "2020-07-02T14:53:07Z")

</div>

I might be misunderstanding, but does CmdStanPy’s `output_dir` argument do what you need? (In case that’s not what you need I’m tagging CmdStanPy expert @mitzimorris)

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 2:57pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/3 "2020-07-02T14:57:13Z")

</div>

what @jonah said - you should be able to create unique output directory names via the ‘output\_dir’ arg to the `sample` method. alternatively, once you have the CmdStanMCMC object, you can use the “save\_csvfiles” method.

---

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 5:03pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/4 "2020-07-02T17:03:03Z")

</div>

Given the nature of distributed file system I am working with, specifying directories with `output_dir` or `save_csvfiles` doesn’t solve my problem. The workflow management system (HTCondor) copies over my stan executable and input data to the local scratch drive for faster IO, so naming a directory via `CmdStan` means the program looks for that dirname in local scratch not my home directory on the submit machine.

Would either of the `CmdStanModel` or `CmdStanMCMC` objects contain the name of the streamed output csv file or I guess the components to reconstruct it without having to grep a directory?

---

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 5:04pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/5 "2020-07-02T17:04:12Z")

</div>

Wanted also to say, I **love** the `CmdStan` + Py/R wrapper design!

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 5:06pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/6 "2020-07-02T17:06:37Z")

</div>

> [@mtwest](#):
>
> Would either of the `CmdStanModel` or `CmdStanMCMC` objects contain the name of the streamed output csv file or I guess the components to reconstruct it without having to grep a directory?

yes, the CmdStanMCMC object’s ` __repr__ ` function will print the names and locations of the output csv files.  
so `print(<obj>)` should work, no?

you’ll have to parse the csv\_file names from the output…

---

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 5:54pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/7 "2020-07-02T17:54:06Z")

</div>

> [@mitzimorris](#):
>
> CmdStanMCMC object’s ` __repr__ ` function will print the names and locations of the

Yeah I see how parsing that is a bit of a pain. Doable but not pleasant.

```
(Pdb) fit. __repr__ ()
"CmdStanMCMC: model=schools chains=1['method=sample', 'num_samples=2000', 'num_warmup=1000', 'save_warmup=1', 'thin=1', 'algorithm=hmc', 'adapt', 'engaged=1']\n csv_files:\n\t/m/home/home2/21/westm1/unix/projects/eight_schools/schools-202007022027-1.csv\n output_files:\n\t/m/home/home2/21/westm1/unix/projects/eight_schools/schools-202007022027-1-stdout.txt"

```

[runset.set\_csvfiles()](https://cmdstanpy.readthedocs.io/en/latest/_modules/cmdstanpy/stanfit.html#RunSet.save_csvfiles) is the function that moves the files from `/tmp` to the given directory. Would it be acceptable, @mitzimorris, for that function to take an additional optional argument, ala `new_filename`, so that one could change StanExeName-DateTime-Chain?

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 6:09pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/8 "2020-07-02T18:09:57Z")

</div>

short answer no. chain number is necessary.

parsing a little bit of text is what Python excels at - I know its a pain, but you should write the custom function that suits your needs.

---

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 6:10pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/9 "2020-07-02T18:10:50Z")

</div>

OK, leaving chain number…  
I can write up the PR.

---

<div class="post-metadata">

**Author:** ![jonah](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jonah/32/200_2.png) [@jonah](https://discourse.mc-stan.org/u/jonah)\
**Post date:** [July 2, 2020, 6:12pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/10 "2020-07-02T18:12:46Z")

</div>

I agree that chain is necessary. Other than chain number I think it should be fine to change the names, so I’m OK with `new_filename` if the chain number is still tacked on at the end. @mitzimorris ?

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 6:15pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/11 "2020-07-02T18:15:14Z")

</div>

I don’t think this is generally a good idea, so no, I don’t want to add this feature.  
as I said earlier, this isn’t hard for a user to implement for themselves given their use case.

from the point of view of both usability and maintainability, the fewer features, the better.

furthermore, putting timestamp and chain number in a file name is for the users own good. managing output files is a long-term reproducibility issue - we’re trying to help.  
output is like a lab notebook. do you take off the bar code and tracking number of your samples? please don’t!

---

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 6:27pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/12 "2020-07-02T18:27:09Z")

</div>

I understand the desire to help but forcing users to follow the developers required organizational framework is problematic.

I can solve this problem with parsing `. __repr__ ` as suggested.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 6:35pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/13 "2020-07-02T18:35:23Z")

</div>

The organizational framework addresses two common modes of working with Stan:

- development - run the model over and over, outputs are summarized and visualized, but not saved

- production - the user has a production set-up which is user specific.

A production pipeline should be scripted, so specifying additional arguments isn’t a burden.

We’ve gone with default options that favor learning/teaching/development because this is where most people spent their time working. therefore writing to `/tmp` is the default.

---

<div class="post-metadata">

**Author:** ![jonah](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jonah/32/200_2.png) [@jonah](https://discourse.mc-stan.org/u/jonah)\
**Post date:** [July 2, 2020, 6:45pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/14 "2020-07-02T18:45:34Z")

</div>

I could go either way on this.

On the one hand, I agree with @mitzimorris that we’re trying to provide the options that are safest and make sense in the vast majority of cases.

But I also agree with @mtwest that

> [@mtwest](#):
>
> forcing users to follow the developers required organizational framework is problematic.

But I guess we’re not technically _forcing_ it here (it’s possible to get around it with a bit of work, as discussed).

So I’m OK with leaving it as is, but I’m also OK with adding the (non-default) option that @mtwest is requesting.

In fact, I’m sure that @mtwest isn’t the only person who will encounter this problem. We would like these CmdStan wrappers to play nicely with distributed file systems such as the one @mtwest is using, so if this will be a regular issue for people it seems reasonable for us to address that. Or is this something more unique to @mtwest’s particular setup than I realize?

@rok_cesnovar what do you think?

---

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 6:46pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/15 "2020-07-02T18:46:54Z")

</div>

@jonah, as I marked in my previous reply, I am following @mitzimorris advice and just parsing some of the metadata. This isn’t worth taking up more thread inches. I’d say just drop it.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 6:47pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/16 "2020-07-02T18:47:59Z")

</div>

> [@mtwest](#):
>
> I can write up the PR.

did you mean to say “issue”? all PRs must have an issue.  
hashing this out in the forums is fine too.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 7:01pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/17 "2020-07-02T19:01:36Z")

</div>

> [@jonah](#):
>
> We would like these CmdStan wrappers to play nicely with distributed file systems such as the one @mtwest is using

perhaps I missed something w/r/t to the problem and proposed solution.

- there’s a cluster run, each node puts files to its `/tmp`
- the user wants to have a single output dir with all csv files from all nodes
- `save_csvfiles` doesn’t let you change filename

we could add a “node id” prefix to the existing csv file. I object to wholesale renaming, but prefixing is easy.

@mtwest does this address your use case?

---

<div class="post-metadata">

**Author:** ![mtwest](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mtwest/32/14096_2.png) [@mtwest](https://discourse.mc-stan.org/u/mtwest)\
**Post date:** [July 2, 2020, 7:05pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/18 "2020-07-02T19:05:08Z")

</div>

Yes I meant _issue_. My apologies for being unaware of the history of people producing PRs without consulting the group first about potential changes.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [July 2, 2020, 7:23pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/19 "2020-07-02T19:23:27Z")

</div>

issue filed: [add option to `save_csvfiles` for filename prefix · Issue #254 · stan-dev/cmdstanpy · GitHub](https://github.com/stan-dev/cmdstanpy/issues/254)

> Proposal for argument `prefix` is to preserve information: model, timestamp, chain - I consider this useful for managing the analysis process, which can be very messy. Also, easy to implmenet. Discuss?

---

<div class="post-metadata">

**Author:** ![jonah](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jonah/32/200_2.png) [@jonah](https://discourse.mc-stan.org/u/jonah)\
**Post date:** [July 2, 2020, 7:32pm UTC](https://discourse.mc-stan.org/t/specifying-output-file-names/16345/20 "2020-07-02T19:32:37Z")

</div>

> [@mitzimorris](#):
>
> we could add a “node id” prefix to the existing csv file. I object to wholesale renaming, but prefixing is easy.

I like that idea. @mtwest Would that solve your problem? I guess we would default to not having a node-id but allow it to be specified when calling the `save_csvfiles()` method?

(@mitzimorris On another note, in CmdStanR we use the name `save_output_files()` to match argument `output_dir` and to distinguish from `save_latent_dynamics_files()`, since those are all csv files too. Should I open an issue in CmdStanPy to unify these names or do you want to stick with `save_csvfiles()`?)

[Next page](https://discourse.mc-stan.org/t/specifying-output-file-names/16345.md?page=2)