"Why do people persist in this belief that a kernel is some mystical realm where software-engineering principles don't apply?"

It's not that it's a "mystical realm where software-engineering principles don't apply", it's that - like embedded - it's a domain where you often face tighter resource constraints. Applying the same engineering principles with different constraints can lead to different trade-offs, and ultimately different best practices.