I think this is the key. From a practical perspective, the reason to do an ESS / time limit is because you expect things to take a while, say more than 5 minutes. So from this perspective you could probably make something like 3 minutes be the minimum time that this algorithm would accept as input. This makes sense because in many models I have, it takes 30 seconds to 2 minutes to compile the thing anyway. The people who really want this feature are the ones who are looking at tens of minutes to tens of hours or more.
Second, how bad is it really if you overrun the ESS requested by a bit? Not really that bad. I mean if I ask for ESS = 100 and you give me 131 I’m not going to complain unless that 31 took 20 minutes to several hours to generate. Even then, if without the ESS/time feature I’d have asked for say 1000 iterations because I have no idea what my ESS is going to be and you stop at ESS 131 after 500 iterations, even though I had to wait for those 31 extra, the truth is I just saved a day of computing.
So I think one of the things to do if you try to do this is to be very explicit about the approximate heuristic nature of this calculation so that you manage the expectations of the user.
I’m not sure how best to tune the warmup time, so let’s assume that problem is solved somehow. Next we need to figure out how long to run the iterations. I suggest first looking at the time taken per iteration and figuring out how much budget there is for iterations. So, as each iteration completes you keep track of how much time it took, then after a few iterations (post warmup) have completed you extrapolate linearly to figure out how many samples you can get by the end of your time budget. If this is within a factor of 1.5 or 2 of the ESS, then just run to the time budget.
If the time budget exceeds 2 or 3x the iteration count, you now need to estimate how many ESS you’re getting per iteration. Run until you get iter = ESS requested. You can’t get ESS more than iter count, so you certainly never need to do the ESS count for the first batch of samples. Then after you hit that ESS count, do one estimate of the ESS with your fourier transform etc. Now use iter/ESS(iter) * ESS_requested to estimate the iters you need. If this exceeds the time budget, run to the time budget. If this doesn’t exceed the time budget, run to that iteration and re-do the ESS calculation. If I ask for ESS=100 you’re basically doing one ESS calculation every O(100) iterations at most.
In most cases you’d probably do the ESS calculation 2 or 3 times per run at most until you zeroed in on the actual iter you need to get the ESS requested.