If you have an adversary with physical access, are not all bets off?

It seems like an adversary who could swap your rng could install a radio broadcaster between your keyboard and your pc or reboot to a rootkit.

Oppositely, you might paint your rng with sparkles or other unique paint schemes to mitigate that particular switch risk but then you'd still have the other possibilities.

Anyway, I don't think this kind of physical spy versus spy stuff is the main risk a custom rng is intended for.