Cool, then my suggested GP(s) to capture structure in the day-to-day intercept applies.
I’m unclear if you mean my original model, or “typical models”, but regardless, if you are fundamentally interested in accurate & optimally-informed inference on the A/B difference, I think it would be folly to a priori assume all variability across days reflects unstructured random effects. For clarity, think of a simpler AB scenario with just one Gaussian outcome, wherein a “all variability across days is random” model would be:
data{
int n_days;
matrix[n_days,2] ab ;
}
parameters{
real intercept ;
real difference ;
real<lower=0> noise ;
}
model{
// priors
intercept ~ ... ;
difference ~ ... ;
noise ~ ... ;
// likelihood
ab[,1] ~ normal( intercept + difference/2 , noise ) ;
ab[,2] ~ normal( intercept - difference/2 , noise ) ;
}
And a model permitting GP structure to the intercept as I’m suggesting would be:
...
parameters{
vector[n_days] f_intercept ;
real difference ;
real<lower=0> noise ;
... // GP-related parameters here
}
model{
// priors
f_intercept ~ GP(...) ;
difference ~ ... ;
noise ~ ... ;
... // GP-related parameter priors here
// likelihood
ab[,1] ~ normal( f_intercept + difference/2 , noise ) ;
ab[,2] ~ normal( f_intercept - difference/2 , noise ) ;
}
Take note that the GP is on the intercept, i.e. the quantity reflecting the mean outcome regardless of the AB condition. I think it’s certainly the case that, especially in the realm of human behaviour, temporal structure abounds and burden of proof would be on accounts positing absence of said structure. And certainly if such structure exists, the second model permits attribution of the variability associated with that structure to the GP, decreasing the left-over variability/uncertainty that must then get dispersed among difference and noise. The first model has no such mechanism and thus difference and noise both will be left with greater uncertainty.
It’s even possible (& possibly advisable, depending on the context) to add structure to capture structured by-day variation in the AB difference too.
The key to all of this is to employ a principled workflow whereby:
- domain expertise informs on the qualitative structures included in the model (i.e. decisions on whether to include a GP across days on the intercept or not; for such binary decisions I’d lean toward a low bar for inclusion)
- domain expertise informs on key parameters of any included structures, including where possible parameterization that permits simple specification of structural simplicity-preferring priors (ex. parameterization of either model above permits a peaked-at-zero prior for the AB difference, where a zero difference reflects a simpler “less structure” model; in the case of the GP, a prior on noise to be not-peaked at zero and a prior on the GP aplitude that is peaked at zero similarly reflects a simplicitly-preferring parameterization-&-prior combo).
- Both prior predicitive checks tand posterior predictive checks to validate implmentations of 1 & 2 are actually consistent with both domain expertise and the data at hand.
