How to compare different algorithms for Bayesian inference in terms of speed (and subject to effectiveness)

It would be good to careful with definitions. Effective sample size is a theoretical concept that can be estimated with or without knowing the truth (some of the figures in this thread N_eff BDA3 vs. Stan - #21 by avehtari show both). Thus, please be more specific to which estimate of ESS you are referring to.

As discussed e.g. in BDA3 chapter 10, if we want to estimate the posterior distribution of theta and we would get S independent draws then the scale of uncertainty of theta is sqrt(1+1/S)sigm, it really doesn’t help us to
have large S, and correspondingly with dependent draws large ESS. This is useful to remember when comparing to distributional approximations which in best cases have very small RMSE which would correspond to really high ESS (and analytic solution would have infinite ESS). Naturally this makes generic comparison more complicated as we really should report then how these estimates are going to be used and which error matters.