We are updating sbc package and wishing for some feedback!
If ecdf_diff is a 45-degree rotation (i.e. diagonal subtracted) version of ecdf, as explained in this Aki’s lecture, I was wondering whether ecdf_diffcould have flat areas like the following plot. What do you think?
Is there any way to make it flat or is it natural for ecdf_diff to not have flat areas?
We have used
aes(x=x, y = y-x)which resulted in this.
This is the code for ecdf_diff. Feedback is always welcome!
@bnicenboim @jonah @paul.buerkner @avehtari
1 Like
I’m helping @hyunji.moon write the library and would also like to chime in on how we should carry this out.
What are the things that this new package can do but the current form (sbc function in rstan and cmdstanr) cannot?
A few tradeoffs. Since prior and posterior samples are provided from the user and an object is made from them, it makes it easier to create plots/metrics/diagnostics with a well-defined structure. This will help with maintenance in the future, and to users, who no longer have to follow variable naming rules in stan files but just provide samples. However, what I’m concerned is it increases the amount of “work” the user has to do, since he/she will be responsible for generating samples.
Receiving samples in this form will also help by removing the need to update individual sbc functions in each stan library, but instead have one library that can accept samples from any library. Hence a single point of maintenance.
Roadmap?
After core algorithms are implemented, polishing the library and checking to make sure both samples from rstan and cmdstanr work is a priority.
Since new SBC algorithms are being published, and also some that we’re considering to research and implement, additional algorithms could be added once the library is stable.
Finally, I would personally like to attempt to make the sampling process automatic. The user would only have to provide a StanModel or code as input. Or alternatively receive formulas like brms. This is a long term goal that isn’t a priority right now, given that it would take considerable amount of work.
Would the following be possible? I(@hyunji.moon) would like to use this as an opportunity to continue research on sbc approximation (and prior sensitivity) and not put too much effort into interface (hope to concentrate on computation, approximation)
Since I’m aiming for cross-library compatibility and ease-of-use, I think library design and interface is equally important. There will be design decisions we’re gonna have to make along the way, which is best discussed with folks who have more experience than me. That said, how polished does this library need to be, if we’re gonna publish it?
2 Likes
Hi,
Sorry for the silence, I’m quite busy at the moment.
But I would say that if you want to get proper feedback on the code and functions you should already set it up as an R package. This way you can include proper documentation to the functions that already exist. And ideally a vignette.
Maybe check https://r-pkgs.org/ and https://usethis.r-lib.org/ ?
3 Likes
