# Getting environment variables

**URL:** <https://discourse.mc-stan.org/t/getting-environment-variables/17645>\
**Category:** CmdStan\
**Tags:** user-defined-functions\
**Created:** [August 25, 2020, 7:12am UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645 "2020-08-25T07:12:26Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [August 25, 2020, 7:12am UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/1 "2020-08-25T07:12:26Z")

</div>

Sometimes it’d be useful to pass integers or reals to a CmdStan program which are somewhat independent of the data set, e.g. grain size for `reduce_sum` or `rel_tol` for the ODE solver. I wonder what others think of the idea of utility functions like

```stan
int get_env_int(string key);
real get_env_real(string key);

```

but the parser doesn’t like `string` as a type so I resorted to getting just one variable,

```
#include <cstdlib>
int
get_grainsize(std::ostream* pstream__) {
  const char* str_grainsize = std::getenv("STAN_GRAINSIZE");
  return atoi(str_grainsize);
}

```

I wondered if strings will be added some day to the parser? Does it seem like a bad idea generally to create another way for variables to enter the model? I still think there’s a case for “dev” tunables like this.

---

<div class="post-metadata">

**Author:** ![rok\_cesnovar](https://avatars.discourse-cdn.com/v4/letter/r/7bcc69/32.png) [@rok\_cesnovar](https://discourse.mc-stan.org/u/rok_cesnovar)\
**Post date:** [August 25, 2020, 7:51am UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/2 "2020-08-25T07:51:16Z")

</div>

> [@maedoc](#):
>
> I wondered if strings will be added some day to the parser?

If this were added in the parser, we could support a constant string, much like we support it for print(). I was just working on this for a profiling prototype, where we require profile section names. Its not as easy as adding a regular Stan Math function, but not a huge deal as well.

One thing that is problematic wrt to environment variables is that they can be modified during runtime. But I guess if someone does that, its their problem.

How about

```no-highlight
./model sample data file=data.json runtime_settings grainsize=100 b=2 c=3 

```

There are probably better names then `runtime_settings`. grainsize, b and c would be added to the var\_context and passed to the model constructor as if they were a part of the data file. If we limit this to scalars (which I guess is the actual use case), this should be doable.

When you are working on a cluster in the shell and just want to update a single number (like the grainsize) it can be very annoying to open a data file and change it (more so if the data file is big, which is usually the case when you resort to using a cluster or an AWS instance), so I would support any resolution to this problem.

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [August 25, 2020, 9:12am UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/3 "2020-08-25T09:12:14Z")

</div>

I thought of passing as command line arguments as well but not sure how much work that is. Supporting both (with cli args overriding env vars) would cover all scenarios (HPC, CI pipelines, user scripts) that I can imagine, with functions like the above.

I hadn’t thought of the environment changing while a process is running, but I can’t imagine a situation where it’s useful, since for such tuning purposes, it’s the final summary stats (neff/s) per tunable value that should be used.

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [August 25, 2020, 1:46pm UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/4 "2020-08-25T13:46:37Z")

</div>

> [@rok\_cesnovar](#):
>
> grainsize, b and c would be added to the var\_context and passed to the model constructor as if they were a part of the data file

On second thought, I would just drop the idea of env vars for now, since including some of the `data { }` vars on the command line would be sufficient (env vars can be passed as args without problem), and avoids problems like what to do if the env var isn’t present.

Lastly, it might be easier to implement as a single csv argument,

```no-highlight
data file=foo.json values=grainsize=100,b=2,c=3

```

since it seems like the argument parser constructs the parse tree prior to doing the parsing, without a varargs mechanism.

_edit_ I tried to start implementing and reading the argument parser code is easy but I have don’t really understand how the `var_context` objects figure out what’s `real` and what’s not before getting passed to the model constructor.

---

<div class="post-metadata">

**Author:** ![rok\_cesnovar](https://avatars.discourse-cdn.com/v4/letter/r/7bcc69/32.png) [@rok\_cesnovar](https://discourse.mc-stan.org/u/rok_cesnovar)\
**Post date:** [August 28, 2020, 6:52am UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/5 "2020-08-28T06:52:37Z")

</div>

You would have to add functions `add_int_scalar()` and `add_real_scalar()` to var\_context and then call those in command.hpp of cmdstan.

Like for example (this is hardcoded, just to show the point):

```cpp
//////////////////////////////////////////////////
  // Initialize Model //
  //////////////////////////////////////////////////

  std::string filename(
      dynamic_cast<string_argument *>(parser.arg("data")->arg("file"))
          ->value());
  std::shared_ptr<stan::io::var_context> var_context
      = get_var_context(filename);

  // new code
  var_context->add_int_scalar("N", 5);
  var_context->add_real_scalar("d", 1.2);
 // end of new code

  stan::model::model_base &model
      = new_model(*var_context, random_seed, &std::cout);

```

I threw something together quickly for var\_context though it does yet work: [https://github.com/stan-dev/stan/compare/add\_scalar\_to\_var\_context](https://github.com/stan-dev/stan/compare/add_scalar_to_var_context)  
It does increase the size of the vars maps but then they dissapear. Probably missed something subtle, but I think this is close.

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [August 28, 2020, 11:58am UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/6 "2020-08-28T11:58:57Z")

</div>

My reading of the var\_context and the json and rdump implementations was that var\_context provides the generic interface but not the storage. I think the missing piece in your code is to have the methods which return the list of data to first look in the maps you added before looking in the data storage read from the file.

Another way to structure it would be to have a second var context which has the cli settings and a final var context object which knows how to merge the two sources of variables. This seemed a little over complex though so I left it on the back burner, and it would be easier to have a working proof of concept based on your changes.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [August 28, 2020, 6:40pm UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/7 "2020-08-28T18:40:46Z")

</div>

> [@maedoc](#):
>
> My reading of the var\_context and the json and rdump implementations was that var\_context provides the generic interface but not the storage.

var\_context assumes that variables are either real or int - hence all methods have suffix `_r` or `_i` -  
cf: [https://github.com/stan-dev/stan/blob/1abc0f3bae87843d26cd4b20f708cc78801ad02e/src/stan/io/var\_context.hpp#L42](https://github.com/stan-dev/stan/blob/1abc0f3bae87843d26cd4b20f708cc78801ad02e/src/stan/io/var_context.hpp#L42)  
and  
[https://github.com/stan-dev/stan/blob/1abc0f3bae87843d26cd4b20f708cc78801ad02e/src/stan/io/var\_context.hpp#L76](https://github.com/stan-dev/stan/blob/1abc0f3bae87843d26cd4b20f708cc78801ad02e/src/stan/io/var_context.hpp#L76)

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [August 28, 2020, 6:48pm UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/8 "2020-08-28T18:48:27Z")

</div>

> [@maedoc](#):
>
> _edit_ I tried to start implementing and reading the argument parser code is easy but I have don’t really understand how the `var_context` objects figure out what’s `real` and what’s not before getting passed to the model constructor.

this is part of the implementation - for Rdump format, we have our own ad-hoc parser - [https://github.com/stan-dev/stan/blob/develop/src/stan/io/dump.hpp](https://github.com/stan-dev/stan/blob/develop/src/stan/io/dump.hpp) and for JSON we’re using the rapidjson parser.

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [August 28, 2020, 10:15pm UTC](https://discourse.mc-stan.org/t/getting-environment-variables/17645/9 "2020-08-28T22:15:59Z")

</div>

my question/confusion was really about how should we be able to add variables to the var context, since this is just an interface (I.e. the documented methods are all virtual). I wouldn’t implement it for each file type since we are talking about variables not passed as a file but as command line arguments.
