The only other place we could keep them would be in the TPM, but that's slow to access and would adversely affect boot times in normal mode

Has this delay been quantified? I've used Linux-based systems with dynamic root of trust, with the TPM being queried at boot. There wasn't any noticeable delay in boot.