A blog is the most simple to scale application, just one step after the static content. The fact that wordpress has traditionally been not very scalable always used to puzzle me...
As a PHP dev with ~100 WP installs under my belt and plenty of customization, I think I'm qualified to say that WordPress isn't written to be scalable. It's actually kind of crap. Many of the things it does to make writing plugins easier for newbies are Very Bad Things in PHP. WordPress is a memory hog, to the point that foreaching over query data in the wrong way can cause you to hit the memory limit, even if just unwinding your foreach into a copy/paste wouldn't. The memory leaks are somewhat nonsensical, and they make scripting with the WordPress API a minefield.
I'm not an expert on WordPress internals, but the scene is definitely ripe for a replacement simple due to the quality of the API. WordPress has been good enough for most people for a long time, but it has many weak points.
IMHO the problem is that's old technology, who used to write free software N years ago is now busy doing startups ;) So the "next generation" of free software web stuff is missing in part.
Whats more puzzling is the need to render things from the sever when you just want to show people posts, which are really just static stuff. Thats why I love Jekyll.
Caching produces confusion, is my guess. You have to know it's there to realize how to fix the "oops, I made changes and they aren't showing up" issues.
Should be, but Wordpress is built to accomodate everything, and a caching layer baked in by default would screw up so many plugins, half the community wouldn't know what to do with their blogs.
If it was baked in from the beginning, no problem, but there is so much plugin momentum in Wordpress now that throwing a caching layer on top of it would confuse the hell out of a lot of people who just want to write words.
