Analysis API


#41

Yes.

  • in RStan and PyStan: the draws of parameters within a chain is a contiguous block of memory that can be accessed with double* (see @ariddell’s comment)
  • RStan and PyStan are able to handle exceptions (@ariddell’s comment), but this may not be the best way to handle output (@betanalpha’s comment)
  • this was the proposed \hat{R} function signature (for a single parameter):
  • @betancourt is proposing a parallel set of functions for checking diagnostics that returns a boolean and optionally passes a std::string message to a writer:

#42

Thanks for the links.


#43

It’s pretty much either src/stan/analyze/mcmc or src/stan/mcmc/analyze. I’m OK either way. Here’s what’s there now:

ls src/stan/
callbacks/    io/           mcmc/         optimization/ variational/  
command/      lang/         model/        services/     version.hpp   

Will there also be services methods for these?


#44

I think we were all thinking these were the service methods, but I could be mistaken.


#45

I’m starting on the output API signatures and just re-read this thread. It took some effort to catch up so I’ll summarize the thread and make some suggestions:

  • All the interfaces can handle exceptions, we use exceptions, so we should feel free to use them in the API.
  • Many of the diagnostic functions return complex warnings and we want to convey those to users in a standard fashion which suggests being able to produce a string summary and pipe it into a writer along with the actual return value of the diagnostic.
  • Failure in a diagnostic is not the kind of failure typically conveyed with an exception because diagnostics sometimes fail, it’s what they do.
  • Michael solved this problem by having two flavors of the same function: a compute variant and a check variant which differ in their return type (either the statistic or a bool). I think this would work fine but an interface might want to first compute a set of diagnostics and store them, then later only produce warning/error messages.

I think the thread so far (not sure where this went in meetings, wasn’t able to make recent ones) suggests we are mixing up the C++ and services responsibilities in a way that’s not helpful.

Suggestion for the C++ API (just using ess as the example):

  • A function ess_type compute_ess(...) that simply calculates the diagnostic;
  • A function ess_analysis_type analyze_ess(ess_type& ess); that creates the standard summaries (including the text summaries) and returns an object that contains them
  • Go ahead and use std::vector<double> it makes the interface cleaner and these functions will be called from C++ so it’s fine.

Suggestion for the services API (just using ess as the example):

  • A function double ess(ess_type& ess, writer& writer) that returns the ess as a double value (could also just see a void return type). The writer type must be capable of handling ess_type.
  • A function bool check_ess(ess_type& ess, writer& writer) that returns true for whatever we consider “passing”. The writer type must be capable of handling ess_analysis_type
  • Use std::vector<double*>/std::vector<size_t*> otherwise interfaces that are providing parameter values from their side to Stan will need to construct the std::vector<double> for every call to a diagnostic method

This breaks up the code into a few more parts which I realize carries a cost but 1) it keeps the writers out of the Analysis API (since we’re talking about a C++ API that’s good); 2) allows the interfaces some flexibility about whether they want the values or the analysis of the ess; 3) Provides the opportunity to standardize what the interfaces will see as a given “analysis” (e.g.- effective sample size) by isolating those into ess_analysis_type; and 4) gives the interfaces a chance to decide if they want to treat the output differently based on the check without needing to inspect the diagnostic itself.


#46

It looks like this is the critical design driver. Which makes sense—it’s why we were trying to pull things into shared service functions.

I very much like the idea of having string summaries consistent across the interfaces. Is there an easy way to do that and let interfaces determine things like line breaks? I realize that’s a detail, but just wanted to note it before moving on.

We should also do that in our “check” functions in the Stan math library. Those just automatically throw exceptions. @syclik and I originally set things up with all kinds of fancy traits control a la Boost, but I think having the basic check function is nice. The problem architecturally there is that there are things like checks for positivity, but when you pass a vector, you can’t just check for positivity, then throw a message, because the message should indicate the first element of the vector was found to violate the constraint.

Is this going to be a problem for a simple breakdown into an underlying C++ check and then a wrapping report?

I completely agree. We want to reserve exceptions for truly exceptional conditions that can’t be handled by the function in which they occur. The whole point is just to provide a scope to handle the exception further up (by aborting or fixing things, perhaps logging a warning, and continuing). We absolutely don’t want to start using exceptions to mock up ordinary control flow.

I’m assuming the “analysis API” and “C++ API” the same thing? I like this decomposition into simple C++ functions that compute relevant statistics and services methods that the interfaces can use to access them. It does bring to mind the adage that you can always sell a computer scientist another layer of abstraction.

Even if it doesn’t work for everything, I think factoring the code this way to start with is a big win as the result is two simpler pieces composed together in a natural way.


Check function `compute` vs. `check` variants
#47

At the moment my suggestion is to have the logger signature be:

logger::warn(std::vector<std::string> warning_message);

Then the call in the sampling code can be:

logger.warn({"Error of type X", "Exposition on how wrong things have gone 
with X that does not include any linebreaks and is possibly very long, it's fine 
producing a std::vector<std::string> here because this is not performance
critical", "Nice suggestion about how to fix things."})

Then the interface 1) gets the hint that each element of the message is conceptually separate and has one opportunity to include linebreaks; and 2) can use a standard line-wrapping function to add linebreaks to longer messages as needed

Only in the stan-dev/math check functions, right?

Yes that’s clearer. I wasn’t clear because I didn’t want to start a fight over it. I’m thinking:

  • “analysis API” expects calls from C++, uses things like std::vector
  • “services API” expects calls from C or C++, uses things like std::vector<double*> and writer objects directly.

It would be really cool if we could make that distinction more broadly but we have a big need for logging/writers deep in the algorithms so probably won’t be able to really be consistent there (yet).


#48

Do we ever want line breaks in log functions? It makes the log files very hard to parse. Right now, we’ve just been writing for human readable output, but going to logging we probably want to be more standardized and log-like.

I do think if we’re actually showing things to people in some kind of console, the interfaces will need to choose to either wrap or crop.

I don’t think this split is controversial:

  • Analysis API: simple C++ arguments and returns; clients include the services API and external library users

  • Services API: C++ data structures with writer/logger callbacks for output, clients are the interfaces


#49

Hell no I don’t want that but for some reason it keeps getting brought up as a barrier to cleaning up output. The point was to do something that allows for formatting but doesn’t enforce it.

Yes. We need to make this better, even make parsing unnecessary if we can.


#50

I’m totally OK with making all of our error messages single line. They’re not now, but we can make it a goal.


#51

One thing to keep in mind – if all error/warning messages are passed in triplets (type, description, resolution) or whatever the interfaces could decorate each text differently, so it’s just about line breaks. The question is whether or not that proves too restrictive in terms of the the message structure.


#52

That’s the question I don’t see a way to answer (yet) thus the std::vector<std::string> suggestion (and let interfaces sort out linebreaks).


#53

How does everyone feel about starting an issue and then building out at least the C++ Analysis API for ess?

Attempting a summary: For a single parameter, we’d have double compute_effective_sample_size(std::vector<double*> draws, std::vector<size_t> sizes) located in stan-dev/stan:src/stan/analyze/mcmc/compute_ess.hpp.

Is my reading correct in that the above function would just calculate the ess diagnostic and analysis/checks/writers happen elsewhere?


#54

Yeah, that’s a great first start, and it will be easy to change the directory structure or other issues in the PR if needed. Only hesitation I have is whether we require all of the chains concatenated into one vector as in this signature of do a vector of vectors for the chains.


#55

@roulades, please and thank you! That sounds great!

The signature that @roualdes proposed is for multiple chains. Each chain’s draws are in a contiguous block of memory.


#56

I already have a question specific to ess. Continuing discussion on the issue seems more appropriate. Any thoughts about the difference between n_samples and n_kept_samples is appreciated. Thanks.


#57

All the effective sample size calculations, R-hat calculations, etc., are done only with the kept samples. All sizes that are relevant are those of the kept samples.


#58

Thank you for your response. With that in mind,


#59

In light of Posting more updates in discourse.

Previously in this thread it was decided that we’d get the C++ effective sample size calculation going, then

Since the PR referenced in the previous post has now been merged, I think that means the next step is to integrate this into RStan and PyStan. Neither of these repos have merged a sufficiently up to date stan-dev/stan to integrate this functionality, but as I’m swamped in a busy semester, I’m not trying to push people to do this too quickly. Instead, I’m saying I still interested in working on this, and further hope that we might be able to add the latest efforts on Split-Rhat diagnostic and relative effective sample size.