How to compare the estimation results of least square method and Stan model

Modeling
1

Now I have two methods to estimate the parameters,one using the least squares method and the other using the model built by Stan.For the least squares ,we can obtain the estimated value of the parameters, estimated standard deviation,estimated bias,95% coverage , as well as the mean square error of the model,etc. For the stan model , we can obtain the posterior mean ,se_mean and sd ,95% posterior interval .I have some questions about which indicators to choose to evaluate the advantages and disadvantages between the two models, Could you give me some sugestions?

2

If you really want to make a comparison, leave-one-out cross-validation is a pretty standard metric.

3

could you please provide me with some information? Thank you very much.

4

For each data point you have, fit both models with all the data except that data point, then get a prediction for the left-out data point given its covariates. I’m not super expert in this domain, but I can think of a few things you can do from there:

  • checking calibration: for both models, get the 50% interval on the prediction and whether the left-out point falls in this interval. About 50% of the left-out points should fall in their respective 50% intervals.
  • compare mean prediction error: for each left-out point you get a single prediction error for the least-squares model and a distribution of prediction errors from the Stan model (one for each sample in the posterior). You could collapse the latter to a mean then compute the difference between the two methods, yielding a distribution-across-data-points of prediction error differences that you can describe with a mean, quantiles, etc.
  • %ile : get the %ile of the LS prediction error (possibly as an absolute value?) in the distribution of (absolute?) Stan prediction errors yielding a %ile per data point that you can again describe with a mean, quantiles, etc. You can also plot the ecdf of the %ile values, yielding a what is functionally a QQ plot.
5

You should also post over on Cross-Validated to ask for folks’ opinions on best practices for comparing a Least-squares and Bayesian model there.