The biggest problem with Xorg refactoring is that the entire surface area, every non-static function, is module API and there are closed-source modules everyone relies on (mostly Nvidia).