In other words, like GNU HURD. That design is very very hard to make correct and fast.
Correctness suffers because the UNIX API has all sorts of interactions between different parts. This includes atomicity. It's a bear to get this right with IPC.
Performance suffers because you are unable to effectively share data structures. This too relates to the interactions between different parts of the UNIX API.
Look, there is a reason GNU HURD is slow and suffers from incompatibility. It's a cute thought experiment, dominating academia around 1990, but it's not actually fast or maintainable. Experience has proven this.
I was talking about systems like KeyKOS and GEMSOS fielded in production before Hurd was a thing. Then, systems like QNX, OKL4, BeOS, Minix 3, and others that largely removed performance issues often with self-healing and legacy app support. Actually, BeOS and QNX in Blackberry Playbook outperformed monolithic competitors. These altogether long proved our approach builds reliable, fast-enough systems with more security.
GNU Hurd is some crap along lines of Mach that tried to mix too many models while not leveraging lessons learned by others far as I can tell. It's something I hear about every year without any real field use or evaluation results. It's not representative of anything in microkernels except a bad approach.
GenodeOS is a better example where they apply many lessons from old school with modern components and virtualization. Already proven for embedded with desktops in alpha stage.
No, those do not perform well. They are just less horrible than GNU HURD. When you apply similar optimization and implement similar functionality, monolithic kernels always win. It cannot be otherwise; think about it.
Self-healing is generally a security problem. It gives the attacker a second chance. It's also generally a failure. You might think you can restart, but there are huge problems: Instead of a crash, you may get a memory leak or hang. Hardware may be in a strange state, needing a power cycle to restart. Other things start failing once one driver is down. Most systems are unable to keep DMA from scribbling all over everything in RAM, and probably all are unable to keep it from scribbling all over a filesystem.
"No, those do not perform well. They are just less horrible than GNU HURD. When you apply similar optimization and implement similar functionality, monolithic kernels always win. It cannot be otherwise; think about it."
I do. It doesn't have to be better. It simply has to perform well enough that users accept it. Older systems did that slowly. BeOS was a great example where it was running on 90's hardware several movies, graphic animations, a song, and productivity apps all simultaneously with no slowdown. Blackberry Playbook outperformed iPad in tests I saw in responsiveness with one demo running a 3D game with other intensive apps simultaneously. It's at the point where Linus et al's argument about performance being too limited is ridiculous. Only users maxing out performance with little care of reliability will need a monolithic kernel on COTS hardware.
"Self-healing is generally a security problem. It gives the attacker a second chance. It's also generally a failure."
What are you talking about? There's a bit of extra attack surface due to more code and interactions. The first one, though, was implemented in KeyKOS kernel whose total size was around 20Kloc. MINIX 3's Reincarnation Server is straight-forward, too, given how components it restarts are designed. It's actually easier to get this right than reliability and security of monolithic systems since it's simpler and smaller than them. I mean, starting with UNIX Hater's Handbook and such, it took monolithic UNIX (and Windows too) decades to get where they are in reliability and security. The stuff I push got most of that done in first few years with a handful of people plus acceptable performance. So are you arguing against microkernels getting it done or in favor of throwing 8+ digits worth of labor at monoliths to achieve similar results? Neither look good in face of evidence.
"Tanenbaum is biased. It's time to move on"
Tannenbaum's is most immature system on my list. I could drop everything he's ever said and done while still having others as exemplars for mainframe, embedded, and desktop use that had acceptable or great performance with better security and/or reliability. You must have a beef with Tannenbaum or something. I respect his work & like the one presentation I watched but don't need it to back my claims.
"Hardware may be in a strange state, needing a power cycle to restart. Other things start failing once one driver is down. Most systems are unable to keep DMA from scribbling all over everything in RAM, and probably all are unable to keep it from scribbling all over a filesystem."
That's all interesting except these kinds of systems, esp proprietary ones, have been in the field for years in systems where failure and unpredictability had to be minimalized. They worked as advertised. Security-focused ones also passed pentests and analysis by people who knew what they were doing. These kinds of things are where monoliths, esp UNIXen and Windows, often failed or took crazy amounts of labor. Even the immature MINIX 3 is more reliable than you describe with all kinds of failures at the component level for a system that stays up. Your DMA example shows you're really grasping at straws to fight microkernels with an example that (a) represents a tiny set of failures in complex HW/SW systems and (b) still applies to monoliths with the exact solutions available for both styles.
Btw, the first IOMMU I found was in a system called SCOMP: a microkernel-like system that was first to be certified to high-security after IIRC 5 years of analysis and pentesting. Name one monolithic OS that pulled anything like that off. Don't worry, I'll wait.
By self-healing giving the attacker a second chance, I mean that it allows an unreliable attack to succeed. Consider defeating ASLR or winning a race condition. Each time the service restarts, you get a second chance to attack.
I have done a professional evaluation of a EAL6+ certified microkernel OS. There were plenty of bugs and design flaws (which I can not reveal) and an even bigger problem. To obtain certification, most functionality is left out. The users actually need this functionality though, so they put it in the uncertified code running on the certified OS. The overall result is less secure because each user program drags along a buggy reimplementation of what would normally be OS functionality. BTW, despite the EAL6+ nonsense, they were way behind OpenBSD and even Linux. It was that bad.
I have also been a professional kernel developer for a different microkernel OS. I assure you that maintainability is not a property of microkernels. You poke something here, and it pops out there. Good luck tracing out why, and good luck making any serious changes to the OS. The reason is that microkernels are deceptive. The individual components are simple, but they have very complex interactions. Glue isn't free. Compared to that, even Linux is trivial to understand and modify.
"Consider defeating ASLR or winning a race condition. Each time the service restarts, you get a second chance to attack."
I considered it. Those problems are handled by eliminating that problem with other means. Input validation, pointer/buffer/array protection, and so on are a start. Restarts are mainly for hardware faults or problems from lingering state. The concept was field-proven for reliability down to CPU level by a certain vendor whose systems ran NonStop. Many others at various levels, esp app's. Recently, academia showed it with "micro-restarts" paper cataloging problems that built up at every layer while showing component restarts knocked out a good chunk with imperceptible downtime. One of my own designs leverages what you describe in an instrumented system to automatically taint and trace execution after components restart enough. Idea being the failed attacks will take me right to vulnerability and patching it. This is only on paper but CompSci teams did similar things in stuff they built.
"BTW, despite the EAL6+ nonsense, they were way behind OpenBSD and even Linux. It was that bad."
I keep hearing these things. It wouldnt' surprise me if it were true given how I called out one vendor over mislabeling what was certified and not mentioning extra untrusted code. Forced them to change their website. Probably same as assholes given there's only so many EAL6+ kernels out there. ;)
Yet, what analysis and pentesting I've read of such assurance activities dating back to 60's shows they deliver results. We have even more methods today. Whereas the CVE's and severity I get out of low assurance software are laughably bad. It might be true that modern vendors are bullshitting through evaluations. Says more about evaluation politics than the methods used: they only work if applied for real. I endorse the methods most of all, old and new.
Btw, the latest from CompSci aiming at EAL7+ is seL4 kernel. The source code for that is available. Feel free to find their vulnerabilities and show them where their models/proofs were inadequate. Whatever you find will factor into other efforts. If you find little, that would be a testament of itself, yeah? I'm neutral as I'm interested in what exact metrics will be for a ROI analysis.
"The users actually need this functionality though, so they put it in the uncertified code running on the certified OS. The overall result is less secure because each user program drags along a buggy reimplementation of what would normally be OS functionality. "
I agree with that one on security front. This often happens. That's why I push for standardized, core functionality in them. QNX and BeOS were again great examples there although not designed for high-security. GenodeOS is doing clean-slate stuff and pulling in components from UNIX land. They're security focused. So, there's potential there. QNX could conceivably be redone for real security but it's not likely to happen. This is a social problem more than technical. A real issue with barebones stuff but not fundamental.
"I assure you that maintainability is not a property of microkernels. You poke something here, and it pops out there. "
"The reason is that microkernels are deceptive. The individual components are simple, but they have very complex interactions. Glue isn't free."
Yes, these are totally true. It's why you need different tooling for debugging them. My old technique was modelling the software as a monolith in source with bug prevention or hunting using same techniques as finding concurrency errors in shared thread and/or actor models. You can also use taint-based methods that track things through the system live or virtualized. Tannenbaum and Hansen had some other methods. Quite a few out there in CompSci and industry.
Yet, you are in for a world of hurt if you try to debug them like you do a monolith esp with tools designed for monoliths. I have a feeling that's what you were doing. I'm not saying there's a lot of publicly available tooling plus guides on it where you'd have had it easier. This stuff, like high-assurance vs mainstream, tends to silo up with knowledge getting obscure/lost and tools getting dusty. The tricks are prevention by your resource sharing and/or middleware plus tooling that models and tracks flows in distributed systems w/ easier subset of its assumptions.
We get this complaint enough that I think I'll try to dig up a collection of tools or methods from CompSci and proprietary sectors to recommend or further develop into something widely available. If I can find time that is. Got many projects I'm working on outside a demanding job. It needs to be done, though. No valid excuse for us hearing this in 2016 without a Github link to reply with except bad priorities among microkernel community.
"Compared to that, even Linux is trivial to understand and modify."
You're the first to ever tell me that lol. I've seen many people give up on high-assurance UNIX/Linux, even significant architectural changes, because of too many difficulties. Largely tight coupling and legacy effects. So, they ended up working at hardware, compiler/language, or microkernel levels to solve issues. Managed to get them solved in believable ways. Makes me think Linux wasn't so trivial. Peer review will tell over time if each issue was really solved. Meanwhile, Linux today has most of the problems it had when I reviewed it 10 years ago. More reliable and usable than before, though, with it only hosing my packages and freezing my desktop every few months instead of days. The backups and restores work great, though. ;)
"Most systems are unable to keep DMA from scribbling all over everything in RAM, and probably all are unable to keep it from scribbling all over a filesystem."
In what context do you mean? When would this occur?
That was burfog's comment I was quoting. It refers to the fact that direct memory access by some devices can bypass any OS or software protections. Breaks whole security model as memory can change arbitrarily. So, risk of attacks or leaks should be mitigated there.
A few methods follow:
1. Use non-DMA links.
2. Use trusted hardware/firmware that mediates things properly.
3. Use IOMMU to enforce access controls on DMA.
4. Use a combo of full safety in system and careful API for access to DMA features.
I used 1 and 2. A few use 4. Number 3 is most common with basic version mainstreaming. Not enough, though, as complex firmware and OS's still provide attack opportunities.
Note: There's also interrupt floods and other esoteric issues to counter. So, it's the start rather than full solution. EMSEC issues too with malicious peripherals.
thanks for your clarification. IOMMUs are a standard part of most motherboards chipsets these days, what would be the reason for not taking advantage of that and using them?
It depends if you trust those chips or not to start with. Not having the weakness, as in trustworthy I/O, is always superior to a tactic attempting to stop the weakness. The other issue is whether mere restrictions on memory accesses will help. In software, microkernels isolating where or who you could talk to were only a start: attacks could happen via a series of compromises. Failures too in the Byzantine failure model. Plus, with monolithic OS's, you can get full control of the system with kernel-mode attacks that further facilitate DMA attacks where the main system looks unmodified in most operation while malware is in peripheral firmware.
Many issues due to how things connect and store information as burfog said. There's not just one thing. Even my microkernel recommendation is only for the start of the software part of a secure system. Actually, trustworthy CPU, ROM, bootloader and drivers are the start if we're being technical. ;)
Every example of a "fast" microkernel has either ripped out expected functionality (debug traces for example) or simply been the first to make an optimization that can be applied to monolithic kernels as well. Fundamentally, microkernels are slower. A bit of thought should make it clear that this can not be otherwise. No matter how fast you can pass a message, it's still faster to not pass a message at all. Also, the overhead of TLB misses when changing MMU mappings is huge. Microkernels can only win when they compete against badly-optimized monolithic kernels and there is no technique that can get past this fundamental truth.
It's an architecture problem. What makes QNX fast are a few
basic design decisions:
- The basic interprocess communication mechanism works like a synchronous subroutine call - you call, you wait, you get data back. Most slower microkernels have unidirectional I/O as a primitive.
- This is very tightly integrated with CPU dispatching, so that calling a service which isn't currently busy is just a context switch, not a full pass through the CPU dispatcher. This and the above are what make QNX fast. If you do interprocess communication by writing to a socket without blocking, then wait for a reply by reading from one, it takes several extra trips through the CPU dispatcher to call another process. Worse, every such call can put the handoff to the new process at the end of the line for CPU time. If you're CPU bound, this kills performance, in some systems by orders of magnitude. The ability to toss control back and forth between processes at high speed is essential. (This is where Mach blew it.)
- Userspace programs can be placed in the boot image and loaded at boot time. So can shared code objects. This eliminates the temptation to put stuff in the kernel so it's available early in startup. File systems and networking are all in userspace.
We keep the same IPC mechanism. We compile the filesystem process right into the kernel. Having done this, we can now avoid half of the IPC mechanism. We enter the "microkernel" just once now, instead of twice, and we leave it once instead of twice. Since the filesystem is now in the "microkernel", we don't need to switch MMU state and have a TLB invalidate. This is a huge win. Now let's repeat this design change for the disk driver, the network stack, the network hardware driver, and all the rest. Performance keeps getting better. This, BTW, is pretty much what most Mach systems ended up doing. They became microkernel in marketing only. The final step is to clean up the code, and then you have a normal monolithic kernel.
Let's also look at things from the other perspective. You could add the QNX IPC mechanism into any monolithic kernel. AFAIK, Solaris DOORS might even qualify. Well, there you go. You can move things to use that whenever you are willing to sacrifice speed and maintainability. If this is so good, why haven't people done it? Hmmm.
Performance is great until those things crash my system. The stuff still happens with graphics drivers on my Linux distro's. I know it's not necessary because it doesn't happen on the microkernel systems and even Windows dodges a lot of it with their SLAM toolkit.
"Let's also look at things from the other perspective. You could add the QNX IPC mechanism into any monolithic kernel. "
Congratulations: you've just reinvented security kernels w/ legacy support from the 80's-90's plus modern separation kernels w/ legacy support of the 2000's. Here's an example to support your point that our model is better with microkernels, user-mode drivers, and monolithic API's in isolated partitions:
Even just putting the drivers and a few critical components in partitions can work wonders. That's what Nizza-like architectures like Turaya and Genode do. Their TCB's are many fold smaller than UNIX's with acceptable performance. You don't even notice it with the laptops of commercial ones (eg INTEGRITY-178B, LynxSecure, VxWorks MILS). Plus, there's around a billion mobile phones running OKL4 mainly for baseband or legacy isolation alongside Android or Windows Mobile. Notice how your smartphone is so much slower than older ones that didn't do that? Wait, you thought it was faster and better than the last one? Exactly. :)
IMO microkernels are a dead end, to get better performance you have to shove stuff in kernelspace, while for say an exokernel the opposite is mostly true.
If we're talking tech, then your post couldn't be more wrong given my BeOS and QNX examples. Performance was equal to or better than monoliths of the time. BeOS especially destroyed competition in concurrency performance due to its architecture. QNX runs at hardware speed basically with real-time properties and POSIX support. BeOS disappeared due to Microsoft monopoly with Haiku making a OSS clone. QNX was at $40 million a year in revenue when Blackberry bought it. Green Hills and VxWorks are doing OK, too, with VxWorks making more than QNX per quarter. Both have desktops virtualizing Windows, Linux, etc on microkernels w/ Gbps throughput.
I don't see why we keep getting these theoretical counters given the proven results of microkernel performance in the field. Tell me why microkernels are too slow when they can only do this on 90's era hardware:
Our side produced highly reliable and secure systems plus high-performance systems. It was always done with a small group with little time. The monoliths took a decade and thousands of man hours to do the same. It's up to you people to justify why those hours were well-spent.
"while for say an exokernel the opposite is mostly true."
I've seen the VxWorks code. VxWorks is not a microkernel.
That BeOS demo did not heavily use privileged interactions. Mostly it showed computation which is the same on any OS. The best thing it showed was a process scheduler which was good at giving priority to things that a user would care about. A more interesting test would be serving files or building software.
I think one should be careful not to read too much into CVE numbers. People aren't exactly trying to mess with KeyKOS, Haiku, QNX, and other weird things. Few people want to bother. None of the Linux problems are inherently specific to monolithic design. The best you could say is the you might have a sandbox that makes things more difficult for the attacker. On the other hand, restarting means you give attackers more chances to succeed.
The best thing you can say is a bug in kernel code that hoses my whole system is less likely to happen several times over. Suddenly, hackers or faults have to work through components' information flows. You keep ignoring that in your analyses. Also why I brought up CVE's because it's impossible that the microkernels had as many in kernel mode just by code size. Still plenty to be found in privilieged processes but POLA and security checks are way easier when memory model is intact.
Btw, one person here who wrote about QNX desktop demo mentioned doing productivity stuff while compiles ran in background with no lag. So there's that use case except not for BeOS. The link below will show you BFS was more like a combo of NoSQL DB, files, and streaming server:
Due to its nature, compilation and build systems are about the slowest things you can do on it. I've seen numbers ranging from 2.5x to 20x slower than Linux but they didnt share specs. I'd swap out the magic filesystem for a simpler one if on a development box. BeOS was aimed at creating, editing, and viewing streaming media, though. Did that very well.
Re sandbox more difficult
No kidding! That's the entire point: get it right or make it harder to beat at least. Monoliths on mainstream hardware are amusement parks with free rides and victims everywhere for attackers. Microkernels on COTS hardware and even modular, typed monoliths on POLA hardware are a series of sandboxes with adult supervision during play and movement. Quite a difference in number of problems showing up and damage done.
Re more chances to succeed
You keep repeating this too without evidence. Attackers need vulnerabilities to succeed. They'll know some to use ahead of time or they won't if we're talking OS compromise. A flaw in one module lets them take one module no matter how many restarts. A flaw in two with a flow means they'll get it in first try. This is why you design it so each flow and individual op on them follow security policy.
The only time restarts give attack opportunities is if your using probabilistic tactics (eg ASLR) or they're waiting for intermitent failure (eg MMU errata). Any high assurance system better not exclusively rely on tactics (ever) and should account for latter (eg immunity-aware programming).
All in all, anything you've said about microkernel systems applies to monoliths in various ways. One model just limits system-hosing faults and hacks a lot better. The question is do you want to accept that risk to squeeze out max performance or eliminate that risk with acceptable performance? Microkernels choose risk reduction while mainstream monoliths choose performance.
An exokernel is definitely not a microkernel, one provides abstraction via (usually) processes 'servers' the other one does via a library which is vastly cheaper overhead wise. I do understand that there will be a need for some sort of IPC just not to the extent of a microkernel.
A microkernel is an abstraction over hardware with minimal code and API. An exokernel is a form of microkernel since it has these properties. It just does thing very differently from most microkernels. Hence a name for that style.
Mach was a microkernel that tried to do a bit too much. Performance and security stayed horrible. Other designs had acceptable to great performance or security. So, it's not representative of microkernels in general despite being interesting research platform back in its own day.
Now, Darwin starts with Mach then basically adds BSD and graphic stack onto it in kernel mode. So, stuff that would have user-mode isolation and performance penalties due to Mach bloat has no penalty but less protection.
So, OS X is clearly not a microkernel system so much as incorporating a microkernel into a monolith. Windows similarly has a microkernel near its foundation for organization purposes I think. Linux is really modular inside similar to microkernels but clearly similarity ends there. So, there's lots of hybrid results where monoliths and microkernels styles are blended a bit for compromise of bensfits.
Can you elaborate on what you mean by "CPU dispatcher"? I am not familiar with the term and have not heard it before. Is this something specific to certain SoC designs? I've never heard mention of this in x86 architectures.
Modern desktop computers are what, a million times faster than counterparts from the 1980's? I will take the TLB misses and reduced efficiency. It's time for safety and reliability to take center stage. Microkernels seem like a great design for that, much moreso than monolithic kernels.
L4Linux proved that this can be done quite effectively, and at the time, it performed better than hypervisors like Xen that were growing in popularity. What GNU Hurd is trying to do is decompose the monolithic UNIX kernel host into smaller services, and that's where the complexity comes in.
If experience has proven anything, is that outside desktop and server OSes, built on legacy of existing infrastructure, micro-kernels rule in the embedded and real time space.
Correctness suffers because the UNIX API has all sorts of interactions between different parts. This includes atomicity. It's a bear to get this right with IPC.
Performance suffers because you are unable to effectively share data structures. This too relates to the interactions between different parts of the UNIX API.
Look, there is a reason GNU HURD is slow and suffers from incompatibility. It's a cute thought experiment, dominating academia around 1990, but it's not actually fast or maintainable. Experience has proven this.