I am curious what kind of workload is prompting you to want this. I've only ever seen stack overflows from predictable recursive errors, which is generally a pretty shallow bug.

Second observation: LuaJIT might be the better platform to pursue this approach. Since the tracer is already on, stack overflows could potentially be another cold path that gets handled on spill; performance impact could be minimal, perhaps none if existing trace guards can be used.