Do you have any experience using POSIX async IO (aio_read, ...) or Linux kernel async IO (io_submit, ...)?
I'm curious in how throughput performance and CPU usage compares with these to SPDK. My (relatively confident) guess is that performance will be better with SPDK, but if it's not that much better programming against a more general interface (a filesystem) is appealing.
Submitting and completing a 4k I/O using SPDK is about 7 times more CPU efficient than the equivalent operation with libaio, which is opening a raw block device with O_DIRECT.
Said another way, on a recent Xeon CPU you can expect to drive somewhere around 3 million 4k I/O per second with SPDK per core. With libaio, you can do about 450,000 per core off the top of my head.
SPDK has no locks or cross core communication required, so it scales linearly with additional CPU cores. Blk-mq in the kernel also helped the kernel scaling problem significantly, but I'm not sure if it is perfectly linear yet.
Most applications need something like a filesystem to function - there is no denying that. Using SPDK requires applications to implement at least the minimal set of features in the filesystem that their application needs. Many databases and storage services already bypass the filesystem or use the filesystem as a block allocator only, so it is not a big leap from there to SPDK.
I'm curious in how throughput performance and CPU usage compares with these to SPDK. My (relatively confident) guess is that performance will be better with SPDK, but if it's not that much better programming against a more general interface (a filesystem) is appealing.