Fitting GP with noisy observations of time

[anon75146577]

@bbbales2

Since the thread has gone this way, the splines assumption that weirds me out a little is if you split the regression into two separate spaces like Wahba does, the non-penalized H0 and then the penalized H1, that leaves you with the improper priors (uniform -inf to inf) on the basis functions that form H0 (cause they’re non-penalized). Does it matter mathematically if I start putting (possibly non-Gaussian) priors on my H0 basis functions? Am I screwing something up? I really liked splitting things out into different spaces to avoid the non-identifiabilities between what you’re fitting and what you’re considering noise.

Nope. That’s exactly what we’re doing here Case study on spatial models for areal data - Poisson CAR/IAR - #69 by anon75146577

A technical point though: H0 are not non-identifiabilities. They’re just directions that aren’t “seen” by the prior. They’re only non-identifiable if you have them in the model a second time (eg if H0 contains constant functions and you fit an intercept + a spline this is non-identifiable unless you constrain the spline to be orthogonal to H0).

@bbbales2

I don’t think a lot of these splines have hyperparameters to marginalize over. Or at least difficult ones like the RBF length scale. Maybe it’s not an advantage because splines just make you think about this earlier (cause you have to specify their domains/decide if you’re going to use an expansion, etc), so the difficulty is just being shifted?

They don’t have a length-scale/band-width parameter. In fact you can get a lot of spline models by letting the length-scale go to infinity. You will often see data try to send you to that limit (or the practical version: length scale > than observed domain), which is another version of the “folklore” form time series modelling that data often tends towards the non-stationary boundary of the parameter space.

Also - in a very soon to be complete paper (next week if the gods are kind) [see also this one] we argue that you need to consider things like domain size when setting priors on the parameters in GPs, so you still need to consider domains. (In a not soon to be completed paper because its two authors are busy, we show that you don’t need to worry about boundary effects - they’re easy to deal with [at least in low dimensions]).