Applying liklihood ratio test to models estimated using LBFGS

Modeling
1

Hi Stan Community,

If I fit a model in Stan using L-BFGS (2 versions, with and without a parameter of interest), if the model has flat priors, is it valid to calculate a frequentist “p-value” for that parameter by using a likelihood ratio test comparing the 2 models?

I’m a stat modelling newbie sorry if this is a dumb question, but it seems to be that this should (probably be the case) but I can’t find the answer to this anywhere…

Thanks!

2

Here is something: http://www.stat.columbia.edu/~gelman/research/published/pvalues3.pdf
and here: https://link.springer.com/article/10.3758/BF03194105

3

Thank you,

I’m trying to estimate about 20,000 models so typical Bayesian approach (e.g. MCMC) seems to be a little too slow. As a practical alternative I had hoped to build a fairly complex model in Stan, estimate the parameters using LBGFS, the try to approximate P-values using a liklihood ratio test (doing this 20,000 times). I know this isn’t a perfect approach, but I was hoping someone could tell me if its reasonable.

I think using Stan to fit LBFGS with flat priors would be essentially equivalent as estimating the model using conventional Maximum Liklihood Estimation? But is potentially a lot more convenient, due to Stan’s nice interface…

4

So the Priors, Posteriors, and P Values section of the Gelman paper doesn’t address this? He mentions: “Greenland and Poole’s proposal to either interpret one-sided P values as probability statements under uniform priors (an idea they trace back to Gossett) use one-sided P values as bounds on posterior probabilities (a result they trace back to Casella and Berger)”.

5

I think the simple answer is just ‘yes, this is fine’. There’s nothing inherently Bayesian about probability…

2 Likes
6

Thanks Charles. Thanks you increasechief for highlighting this section of the paper, its quite intimidating to a non-statistician.

1 Like