I’m not sure what Rasmussen was trying to say, but it’s very much not correct that Bayesian GP models automatically account for this trade off. (As things that aren’t true go, it’s somewhere between “everyone is good at karaoke” and “the moon is made of cheese”)
In the next few days/weeks, @betanalpha will publish a case study about how Bayesian inference with GPs requires some thought. (as well as how to do it)
So I’m not sure that GPs are an appropriate tool for what you’re trying to do. They can probably do it, but it’s “expert level” GP work.
< TECHICAL BIT>
For example, if you want to know if a GP is above a certain level (like 2/sqrt(n), which you would typically use for a pointwise test), it’s not enough to just compute Z-scores (or their equivalent) because you actually have to look at the function at an infinite number of points. The multiple testing correction for this is difficult to work out (the standard method involves very hard geometry). There are more mathematical delicacies as well.
< /TECHNICAL BIT>
Model misspecification makes all of this stuff even harder.
But there’s nothing fundamental about iid Gaussian noise in GP regression. So if you want to use a more appropriate observation model, use it. Mathematically, you can put a GP into almost any position in almost any statistical model. Practically, you need to be sure that there is a lot of information from the data flowing into the bit of the model with the GP and you need to be very careful to avoid overfitting.