> They're the Linux kernel except the desktop versions have a window manager running on top of them that handles the GUI which typically isn't found on a server.
Bingo! Servers are designed to run rock solid stable distributions with the minimum amount of software necessary to get the job done. Desktop distributions come with a desktop environment that allow the users to do things like pick a wifi network or change audio volume. At work we use alpine linux docker images which do not come with even a text editor. They are two very unique use cases. No one sane runs web servers with a full blown Ubuntu desktop distribution because the overhead would be insane. This is one of the reasons why linux servers are so common, and dominate the market share, because they don't have to run full blown desktop environments that'll never be used.
Comparing desktop linux distributions to linux servers is like comparing a ferrari to a honda civic, because both of them are internal combustion engine cars.