X has two drawing APIs. One of these is a part of the core X11 protocol, which is ancient, useless, and nobody uses. The other is the XRender extension which provides modern composite operations, among other things such as gradients. This is what Cairo, for example, uses. X also has font drawing APIs.
The X11 protocol (without extensions) defines about 120 types of requests: create a window, move a window, etc.
Nowadays, there's at least 25 % of them which are useless: usage of server-side font, or the drawing of squares and polygon, are unused by any modern application or toolkit. All of this is superseded by requests from extensions, like the XRender one.
The handling of multiple monitors displays has totally been screwed up. X11 has been designed to work in Zaphod mode (independent monitors). But Xinerama, and nowadays XRandR have replaced it up: recent X servers (released after ~2007) does not support Zaphod mode anymore, even if it's a core piece of the X11 protocol.
Long story short: the X11 protocol (which Xorg/X is an implementation of) was created in 1980, and was made for drawing stuff like this: http://xawm.sourceforge.net/Xaw.gif
The important distinction between X and any modern display server is that nowadays we separate compositor (that which composites application content into what you see on the display) and toolkit (that which draws text boxes, buttons, progress bars etc.) into two. X was designed to do both these things. This isn't practical for several reasons. So far people have worked around this issue, but many feel it's time to stop working around it (since the X codebase is a complete mess due to this "working around"), and develop something from scratch that does what we want for a modern desktop in 2013.
It draws process boundaries in the wrong place then forces enormous complexity trying to deal with the resulting synchronization problems. To be fair, when X was designed, I doubt it ever occurred to anyone that one day you'd have opaque resize, much less a fully double-buffered dance between application, X server, and window manager.
Besides of being too slow to run on current phones, and not natively supporting 3D acceleration, these are the reasons mentioned in the article:
"With respect to shell development (Unity), three major shortcomings of the X stack prevent us from delivering the user experience (f’n’f) we have in mind:
* X shares a lot of system state across process boundaries. This is obviously not a problem in itself but a system-level UI that is meant to provide a beautiful and consistent user experience is likely to require tight control over the overall system state.
* X's input model is complex and allows applications to spoof on input events they do not own. On the one hand, this raises serious security concerns, especially regarding mobile platforms. On the other hand, adjusting and extending X's input model is difficult and supporting features like input event batching and compression, motion event prediction together with associated power-saving strategies or flexible synchronization schemes for aligning input event delivery and rendering operations is (too) complex.
* The compositor hierarchy ends on the session level, and no tight integration into the system from boot time onward is available. For that reason, there is a visible glitch when transitioning the system from a VT-level to the graphical shell level."
Also, it says something when #3 on your big list of complaints about X is that switching graphics modes when starting X is a major problem that needs to be fixed.
It was designed for technologies that are obsolete now, forcing enormous complexity for trivial tasks.
(Disclaimer: I had to work with low level X years ago, so I disdain and hate it with passion).
Today almost everything uses OpenGL acceleration that provides a standard way of drawing things on screen , while being fast and unloading and abstracting work from the CPU.
OpenGL is in itself a meritocracy on drawing standards, companies innovate by extensions and what works in the real world is selected into the API, instead of the complex "by committee" of people that know nothing about drawing design of X.
