> ... which is like DPDK but for storage devices.

honest question, with so many 'things' taking matter into their own hands, how do you ensure that they play 'nice' with each other ?

any and all insights are greatly appreciated !

Two answers:

1) Modern hardware is getting much better at multiplexing resources (aka "making the things play nice with each other"). See for example sr-iov.

2) More and more applications are distributed across many machines, with the resources of each machine entirely consumed by that one application. In that case there is only one "thing" running so the problem of multiplexing goes away. This is the premise of technologies like unikernel.

> 1) Modern hardware is getting much better at multiplexing resources (aka "making the things play nice with each other"). See for example sr-iov.

sorry, but with sr-iov, you still have one single 'control' application muxing resources for applications sitting above it. for example, with dpdk, you more or less take over the complete network card.

the 'control' might be even human in some cases :) which/who carefully lays out the resource mapping.

another example, intel's cmt/cat techniques (https://github.com/01org/intel-cmt-cat/wiki) map nic's rx/tx rings to processors l3 cache etc. this is ofcourse assuming that userland applications have complete control over pci-e lanes etc.

now, if you have two such control applications e.g. one doing packet-io and the other doing disk-io, how do you ensure that these control applications don't starve each other out ?

in canonical settings, kernel would be democratizing (is that even a word ?) access to the underlying h/w. but since that is bypassed, we are in a strange new world...

Sharing CPU resources between user space drivers is certainly a challenge. The best way to view this is that the kernel provides a general purpose solution for sharing resources with some associated overhead. Tools like DPDK and SPDK let you opt out of that, but now you are responsible for intelligently sharing the hardware.

You, as the application developer, have a distinct advantage though - you only need to solve the problem for your application, and using that knowledge can often lead to more efficient solutions. This may mean dedicating cores to the network or disk, or it may mean working in fixed sized batches, etc.

> ... you only need to solve the problem for your application, and using that knowledge can often lead to more efficient solutions...

this ! _exactly_ this :) imho, the fundamental re-architecture of I/O subsystem for x86 machines has kind of relegated this playing field now to mostly solving _only_ s/w problem, rather than a combination of h/w and s/w.

for example, earlier if you wanted to write a very high performance node in, say the epc-core e.g. SGW/PGW/MME etc. you would assemble a bunch of folks with very diverse set of expertise. right from h/w i/o subsystem designers who could do npu's, switch-fabrics etc. to driver dudes, to 'infrastructure' folks to application programmers etc. etc.

in the current incarnation, a vanilla off the shelf x86 machine is more than sufficient. and if your s/w architecture is _right_, you can scale quite easily.

Do you have any experience using POSIX async IO (aio_read, ...) or Linux kernel async IO (io_submit, ...)?

I'm curious in how throughput performance and CPU usage compares with these to SPDK. My (relatively confident) guess is that performance will be better with SPDK, but if it's not that much better programming against a more general interface (a filesystem) is appealing.

Submitting and completing a 4k I/O using SPDK is about 7 times more CPU efficient than the equivalent operation with libaio, which is opening a raw block device with O_DIRECT.

Said another way, on a recent Xeon CPU you can expect to drive somewhere around 3 million 4k I/O per second with SPDK per core. With libaio, you can do about 450,000 per core off the top of my head.

SPDK has no locks or cross core communication required, so it scales linearly with additional CPU cores. Blk-mq in the kernel also helped the kernel scaling problem significantly, but I'm not sure if it is perfectly linear yet.

Most applications need something like a filesystem to function - there is no denying that. Using SPDK requires applications to implement at least the minimal set of features in the filesystem that their application needs. Many databases and storage services already bypass the filesystem or use the filesystem as a block allocator only, so it is not a big leap from there to SPDK.