An X11 server is rarely on the hot path for rendering. Most X11 apps does most of the rendering to shared memory buffers themselves.