Hmm, I don’t even think we try to track this stuff now. We usually find out about overflows by accident, and we haven’t tried to put if conditions around all the things that might blow up. There’s always wishful thinking that functions will work everywhere they should mathematically (and then they don’t). I assume we haven’t tried to build guardrails everywhere because it would take a ton of work and just make models that do run without errors really slow from all the checks.
I think for the most part if there’s a more stable way to compute something, we go for that though. We’d need some sort of more stable reference to know if something went off the rails, and if we had that reference we’d use that.
Yeah this is another thing altogether. Especially if you aren’t sure if the approximation is going to affect anything.
Yeah, I don’t think we could deal with the automatically, but it seems like it might be possible to make it much easier to find these things. Doesn’t really help you short term, but I made a topic to discuss this a little (Debugging overflows in values/gradients).