Subclassing Cmdstanpy

[Funko_Unko]

I don’t want to derail the other thread any further (Cookiecutter-cmdstanpy: a template for cmdstanpy projects sorry @Teddy_Groves1).

@ahartikainen yes, a lot of the functions are mere convenience functions. For the timings eg I do just parse all stdout files and extract warmup and sampling timings if available.

Once I had subclasses for my main functionality the threshold was quite low to add extra functionality as member functions.

[Funko_Unko]

@ahartikainen

Was there any problems?

Is there a canonical way to do this?

Edit: meaning, I have some run of a model, and I want to take off exactly where I left and continue sampling or warming up.

[mitzimorris]

• Restart sampling from a fit object.

replied in other thread here; Cookiecutter-cmdstanpy: a template for cmdstanpy projects - #12 by mitzimorris

[Funko_Unko]

@ahartikainen @mitzimorris

@ahartikainen

Was this after you combined multiple chains? See arviz.InferenceData output, does that look correct?

Edit: fixed relevant quote.

I’ve opened an issue here `CmdStanMCMC`'s function `stan_variable` does not handle arrays of vectors correctly. · Issue #361 · stan-dev/cmdstanpy · GitHub
I did run pip install --upgrade git+https://github.com/stan-dev/cmdstanpy before, so this should be up to date, shouldn’t it?

Edit: It gets weirder. This issue only arises if we take exactly one sample. I’ve updated the issue.

[Funko_Unko]

@mitzimorris

what do you mean by “restart sampling”?

• continue adaptation? in which case, initializing with stepsize and mass matrix should work.
• get more samples? initializing with stepsize and mass matrix, no_warmup, will produce more samples. but there’s a problem in that we don’t save the state of the RNG - the seed + chain_id is the initial offset, but then during a run, we record how many times it’s called. to properly continue, one would need to know both the seed and how far to advance the RNG.

I mean restarting with the current metric, step size and points in parameter space. There were three issues:

• Couldn’t initialize stepsizes with an np.ndarray (this appears to be fixed)
• Couldn’t initialize inits with a list of dicts, added an issue here Can't initialize chains with different inits via list of dicts. · Issue #362 · stan-dev/cmdstanpy · GitHub
• Couldn’t initialize metrics with a list of np.ndarrays or with a single one. Didn’t add an issue.

The last two points aren’t really advertised functionalities of the sample method, but this sure would be convenient (at least for me).

Anyhow, what I did in subclassing the CmdStanModel was just hacking together the appropriate json serialization.

@ahartikainen

In Stan code?

Hm, I do not believe it is possible to infer the size of an array in Stan, or is it? Meaning for example having a data block

int no_elements;
vector[no_elements] v;

and only specifying v? Because I’m lazy, I didn’t want to always type (in python) data=dict(v=v, no_elements=len(v)) but just data=dict(v=v) and let no_elements be inferred from the shape of v. It’s of course not only about the shape of arrays, but still probably not a mainstream functionality.

@mitzimorris

we recommend running multiple chains in order to be able to test convergence. because if you don’t have convergence, whatever your sample is, you have no evidence that you have a set of draws from the posterior, consequently, no reason to believe that the samples are valid.

therefore, you need to wait until all chains have finished sampling. granted, this is problematic if you’re not able to run all chains in parallel or otherwise distribute them across a cluster.

Yes, the use case is not to just discard the aberrant chains and act as if everything was fine. Instead I have the following situation:

I have a reference solution/sample/method, and want to compare it to other methods. For some of the methods, there arise issues during sampling, leading to chains gettings stuck and taking super long. Eventually, they may find the other chains, or they may not. In either case, I wan’t to be able to terminate the chain, and work with the data that I have, to compare the different methods. Currently I only use the timings, but I could also work with either all of the data of the finished chains or with all the complete and partial data.

Things would of course be easier if I would just submit a huge job on a cluster, but, apart from not having access right now, doing things locally and in this way shortens the development iteration times.

@ahartikainen

(source → cpp?)

Just the stan functions/model blocks. See e.g. this Adjoint ODE Prototype - RFC - Please Test - #7 by Funko_Unko or this Adjoint ODE Prototype - RFC - Please Test - #16 by Funko_Unko monster of a figure for an example.

[mitzimorris]

@Funko_Unko

Hm, I do not believe it is possible to infer the size of an array in Stan, or is it? Meaning for example having a data block

this won’t fly because the fact that no_elements is an array dimension is not something that can easily be inferred. from the point of view of the generated c++ program, it’s just another int variable.

also, this:

data {
  int no_elements;
  vector[no_elements] foo;
  vector[no_elements] bar;
 ...
}

when foo and bar have different lengths, where is the problem?

[Funko_Unko]

@mitzimorris

this won’t fly because the fact that no_elements is an array dimension is not something that can easily be inferred. from the point of view of the generated c++ program, it’s just another int variable.

Yes, I was just expressing my confusion at @ahartikainen’s question, because I did not believe it possible (in practice or in principle), because stan does not and cannot know that this int is actually just the size of that vector. I guess such a functionality could be added, but I don’t believe anyone would want to do that. Looks like there would be very little benefit for a lot of headache.

One of the reasons I wanted this functionality is because I have different models, which all can be characterized by some number of main states, which will then induce a model-specific number of auxiliary states and parameters. This can of course be partially done within Stan, but I did not do this for two reasons.

• First, I wanted to keep the stan interface as similar as possible across models, and only change the functions (containing the ODE) and model blocks.
• But also, I believe there were some issues with using inferred data (quantities from the transformed data block) in specifying the size of data arrays/vectors. I don’t know whether there is a way to do this in Stan, and as I had a solution ready, I didn’t bother investigating this further.

@mitzimorris

when foo and bar have different lengths, where is the problem?

There would be no problem, just an exception thrown ;)

[mitzimorris]

@Funko_Unko

But also, I believe there were some issues with using inferred data (quantities from the transformed data block) in specifying the size of data arrays/vectors.

correct, this isn’t possible - variables in the data block must be defined before being referenced, therefore all array dimensions used in data variable declarations must be supplied as data.