Ragged Arrays with reduce_sum?

mathDR · April 7, 2021, 6:40pm

Has anyone figured out a way to use reduce_sum with ragged arrays?

I guess if/when there are “true” ragged arrays, this would be seamless, but is there a hack for until then?

jsocolar · April 7, 2021, 6:48pm

The implementation details will depend on the structure of your data and the computation you need to do.

Here’s one example:
I needed to do a row-wise computation on an array with varying numbers of columns, so I filled the trailing elements of each row with nonsense, and then passed a one-dimensional integer array as data to tell Stan how many elements of each row to grab. It’s embedded in a gargantuan model, but take a look at lines 151, 219, 317, 321, 349, and 742 to see the relevant parts.

github.com

jsocolar/colombiaBeta/blob/master/stan_files/full_colombia_model/occupancy_v6_1.stan

// This is a Stan model for the full Colombia bird dataset, version 6.1
// Changes:   Include spatial effect of subregion, include effect of observer:species
//            Combined variances for spatial effects and taxonomic effects
//            Remove effects of floodSpecialist
//            Remove range-restriction effects in favor of just the "barrier effects"
//            Zero-center all random effects
//            Switch from offset/multiplier to transformed parameters for noncentering 
//              to avoid "sticky boundaries."  See https://discourse.mc-stan.org/t/offset-multiplier-initialization/20712
//            Better priors that interact as desired with effects coding
//            IMPORTANT: data structure changes to effects coding for this model!
//            Removal of offset multiplier soft-constraints (significant reparameterization 
//              probably invalidates some of the old soft-constraints)
//            Better consistent formatting throughout, and slight rearrangement of 
//              parameters and order of partial_sum arguments to keep biogeographic 
//              parameters together in one block

functions{
  real partial_sum(
    // Function arguments:
      // Data slicing and indexing

This file has been truncated. show original

mathDR · April 7, 2021, 6:54pm

ah okay, sure. I originally moved to the ragged array idea because I didn’t want to increase the memory footprint (originally I had filled everything in with zeros to make things rectangular).

Do you think the memory increase vs the reduce_sum parallelization is worth the trade off?

I would think so, right?

jsocolar · April 7, 2021, 6:57pm

If the array is data then the memory footprint shouldn’t matter much (as long as you aren’t running up against the memory limitations of your system). If the array is a parameter then it’s less clear, and probably depends on “how ragged” the array is (how many more elements are in the rectangular array compared to the ragged).

Topic		Replies	Views
Dealing with ragged and partial data Modeling	2	586	December 21, 2017
Problem when converting to a ragged array model (unsure why - syntax? typo?) Modeling specification	6	931	June 14, 2017
Is Ragged Array allowed in Stan? General	20	2629	March 13, 2023
Ragged arrays bug in the language Developers bug	10	1140	May 28, 2021
Ragged array expressions Modeling	17	1633	October 22, 2018

Ragged Arrays with reduce_sum?

Related Topics