An issue here is that mmap and io_uring require fundamentally different software architectures in a performance context. You shouldn’t swap them out.
APIs like io_uring, combined with O_DIRECT, allow you to design your own workload-specific userspace scheduler from first principles. If you are delegating scheduling to a runtime then you’ve forfeited most of the performance advantages those APIs were designed to provide. In many cases, the performance will be worse. By contrast, mmap implicitly delegates all scheduling decisions and it has some advantages if delegation is your strategy compared to a runtime.
The benefits of io_uring are limited without a commitment to designing your own schedulers. On the upside, skilled scheduler designers can increase performance by substantial integer factors using these APIs versus mmap. Designing application-specific schedulers is not easy, it is a high-skill endeavor. But the reward is real.
Using io_uring well requires going all-in on the software architecture it requires to show what it can do.
The real issue here is there's a part 2 (linked at the end of the article) where they get the io_uring implementation to be twice as fast as mmap. So it's a clickbait title for a part 1, which gets resolved in part 2.
So they crunched out 2 articles, one of which is just ragebait. And they both seem LLM written. Maybe they have some cool advice, maybe not... but it's not a format I enjoy reading.
I wouldn't call it ragebait. Trying to optimize things only to end up with worse performance seems to be a rather common outcome. Understanding why that happened is key to overcoming it.
I agree that it should have been a single article though.
I find a lot of LLM articles to be hard to read, but since this one was pretty "straight to the point", I found it to be okay and got the important information across.
The title "The Linux reality check" made me groan, though. LLMs (Claude?) seem to love this phrase.
The problem is not the LLM writing. The problem is the ragebait title which is unsubstantiated, because of part 2. And they know that because they published it in two parts.
This is basically how you write fake news. And you only publish like this for extra clicks.
Rant - CPUs have a cache hierarchy for a reason. We can use smart caches and speculative execution to fetch only the pieces of memory that actually are necessary for computation.
Large parts of the computing infra support this - like RAM, where batching requests is not really helpful, as you can hit full speed with 64 byte accesses.
This means that the tradeoff between bandwidth and latency falls somewhere in the middle, and its worth being smart about it.
MMAP and virtual memory is supposed to be the last level of this, managed by software. But it was designed in the time of spinning disks, where it took 100s of ms to serve a page fault, so parking a faulting process wasn't critical as it was almost certain you couldnt serve a request before the timeslice was up.
RAM was comparatively huge to cache, so it made sense to bring in large chunks at once - in fact this was so true, that if your app started paging, essentially it became unusable.
SSDs changed this math - you can almost certainly serve a request in tens of microseconds even with going through PCIe - hooking up flash directly to the memory hierarchy might make this even cheaper.
So the reason why io_uring exists is because the syscall overhead is now comparable. But this involves a new API and devs having to manage this manually.
I think this is wrong. I'm not sure if this is fixable with the current batch of CPUs, or needs HW support, but imo there needs to be some sort of page fault handling that's much more granular than an entire page (lets say 1-2KB, on the order of SSD page sizes)
We had a similar experience in Polars trying to use io_uring.
Rust isn't inherently bad at io_uring but at least Tokio currently is. I'm not the one who implemented and benchmarked it so this is second-hand information but if I recall correctly Tokio shares one buffer pool for all threads so as you scale to 100+ threads the whole thing grinds to a halt.
Migrating our I/O to a different async runtime than Tokio was rejected. So we'll wait until it's fixed in Tokio and now use regular blocking reads instead.
Rust is just the language they happened to be working in. If you read through the LLM generated garbage it becomes clear that io_uring isnt the problem, their code was just slop / bad.
It would have been more precise to say O_DIRECT instead of io_uring. The point of io_uring is to avoid syscall overhead. What they were after was actually managing a page cache on their own, and it turned out to be more complicated than they thought.
Saying that an application uses io_uring for input/output provides almost zero information. It provides about the same information as saying that an application is run on Linux.
io_uring is an alternative method of invocation for a very large number of traditional Linux syscalls, and the I/O performance that you obtain depends on which syscalls you use and how you schedule them.
You can use io_uring with classic sequential files, in order to perform the syscalls asynchronously, so you can do other processing in parallel with them, without using multiple threads.
You can use io_uring with mmap, in order to obtain much better performance than they got with mmap.
The file readahead that the kernel does when the user generates page faults is not good enough for reaching maximum performance and it has a great overhead.
For maximum performance with memory-mapped files, the user must take responsibility for the readahead, instead of expecting the kernel to do it.
This is done with madvise, but the best results are obtained by avoiding the POSIX-defined advices, which have wrong semantics, and by using 4 Linux-specific advices:
either MADV_PAGEOUT or MADV_COLD are used to inform the kernel about the pages of the file that can be freed because they will not be reused soon, so that the kernel will have available memory where to do file readahead;
either MADV_POPULATE_READ or MADV_POPULATE_WRITE are used to tell the kernel to read ahead immediately some part of the file instead of waiting for page faults, so if these are used at the right times there will never be any page faults, wasting time.
The madvise syscall must be executed through io_uring, so that it will be executed asynchronously.
When not even controlling the file readahead with madvise provides enough performance, one can replace the memory-mapped files with files opened with O_DIRECT, where all the read/write syscalls are done through io_uring.
By using fixed read/write buffers, O_DIRECT removes the overhead of mmap where each time when pages are read from the file the virtual address translation tables must also be rewritten, and the even greater overhead that happens when pages are freed and all the other processor cores must be informed about this (TLB shootdown).
However, to obtain the performance achievable with O_DIRECT, the programmer can no longer rely on the kernel for I/O scheduling, but this must be handled in the application program, which can bring significant complexity.
This means that all the file reads must be launched enough in advance, so that they will be completed by the time the application needs the data, but one cannot launch too many file reads in advance, because that would consume too much memory.
Whenever O_DIRECT is added without rewriting completely how file I/O is handled in the application, that will lead to reduced performance, not to better performance.
> The internet is full of posts declaring io_uring wins over mmap.
Internet is also full of idiots not having done their homework.
Blindly throwing io_uring for mmap and hoping for better perfomance in a highly concurrent environment is a recipe for thread contention and latency spike.
Add to that the kernel lock contention on mmap. Later kernels have tried to increasingly mitigate this. But significant latency on kernel lock contention still exists.
Try mmap + numa like pinning (from software atleast via hashing the work id) and always have the same mmap be used from the same node. This usually yields better latency compared to throwing wonder weapons.
Our database Dip uses mmap burst for opening massive amounts of mmaped kv engine files along with numa pinning of the same mmap kv engine file to the same core so as to reduce cross cache contamination and page cache relocation.
io_uring (available only in more modern kernels) has so far only increased Dip's latency. Hence we have no reason yet for using it. It is a net negative for our usecase.
Use io_uring where appropriate and don't expect every mmap replaced by io_uring to do wonders.
An issue here is that mmap and io_uring require fundamentally different software architectures in a performance context. You shouldn’t swap them out.
APIs like io_uring, combined with O_DIRECT, allow you to design your own workload-specific userspace scheduler from first principles. If you are delegating scheduling to a runtime then you’ve forfeited most of the performance advantages those APIs were designed to provide. In many cases, the performance will be worse. By contrast, mmap implicitly delegates all scheduling decisions and it has some advantages if delegation is your strategy compared to a runtime.
The benefits of io_uring are limited without a commitment to designing your own schedulers. On the upside, skilled scheduler designers can increase performance by substantial integer factors using these APIs versus mmap. Designing application-specific schedulers is not easy, it is a high-skill endeavor. But the reward is real.
Using io_uring well requires going all-in on the software architecture it requires to show what it can do.
The real issue here is there's a part 2 (linked at the end of the article) where they get the io_uring implementation to be twice as fast as mmap. So it's a clickbait title for a part 1, which gets resolved in part 2.
So they crunched out 2 articles, one of which is just ragebait. And they both seem LLM written. Maybe they have some cool advice, maybe not... but it's not a format I enjoy reading.
I wouldn't call it ragebait. Trying to optimize things only to end up with worse performance seems to be a rather common outcome. Understanding why that happened is key to overcoming it.
I agree that it should have been a single article though.
I find a lot of LLM articles to be hard to read, but since this one was pretty "straight to the point", I found it to be okay and got the important information across.
The title "The Linux reality check" made me groan, though. LLMs (Claude?) seem to love this phrase.
The problem is not the LLM writing. The problem is the ragebait title which is unsubstantiated, because of part 2. And they know that because they published it in two parts.
This is basically how you write fake news. And you only publish like this for extra clicks.
There is a pattern in software engineering of:
* Project exists
* New employees come up with idea for efficiency improvement
* Months spent implementing. Old codepath becomes legacy.
* New thing now has extra features bolted on during build.
* Efficiency of new thing turns out worse than original, but now the new features and 'less technical debt' are the drivers.
* New thing launches, old thing deprecated, but there is little real benefit to all those months of work.
Then they fix the new thing implementation[1] and realise the original gains that were promised
[1](https://www.conviva.ai/resource/making-io_uring-actually-fas...)
Second System Effect
https://en.wikipedia.org/wiki/Second-system_effect
Coined by Fred Brooks in "The Mythical Man-Month" in 1975.
Rant - CPUs have a cache hierarchy for a reason. We can use smart caches and speculative execution to fetch only the pieces of memory that actually are necessary for computation.
Large parts of the computing infra support this - like RAM, where batching requests is not really helpful, as you can hit full speed with 64 byte accesses.
This means that the tradeoff between bandwidth and latency falls somewhere in the middle, and its worth being smart about it.
MMAP and virtual memory is supposed to be the last level of this, managed by software. But it was designed in the time of spinning disks, where it took 100s of ms to serve a page fault, so parking a faulting process wasn't critical as it was almost certain you couldnt serve a request before the timeslice was up.
RAM was comparatively huge to cache, so it made sense to bring in large chunks at once - in fact this was so true, that if your app started paging, essentially it became unusable.
SSDs changed this math - you can almost certainly serve a request in tens of microseconds even with going through PCIe - hooking up flash directly to the memory hierarchy might make this even cheaper.
So the reason why io_uring exists is because the syscall overhead is now comparable. But this involves a new API and devs having to manage this manually.
I think this is wrong. I'm not sure if this is fixable with the current batch of CPUs, or needs HW support, but imo there needs to be some sort of page fault handling that's much more granular than an entire page (lets say 1-2KB, on the order of SSD page sizes)
Yay, first article (for me!) that i actually click on because the title is interesting, and then it puts me off because it's LLM generated.
By the way, what does Rust have to do with their problem? Is Rust simply bad at io_uring? :)
We had a similar experience in Polars trying to use io_uring.
Rust isn't inherently bad at io_uring but at least Tokio currently is. I'm not the one who implemented and benchmarked it so this is second-hand information but if I recall correctly Tokio shares one buffer pool for all threads so as you scale to 100+ threads the whole thing grinds to a halt.
Migrating our I/O to a different async runtime than Tokio was rejected. So we'll wait until it's fixed in Tokio and now use regular blocking reads instead.
Rust is just the language they happened to be working in. If you read through the LLM generated garbage it becomes clear that io_uring isnt the problem, their code was just slop / bad.
That was hinted at in the summary for part 2, which i couldn't bring myself to click on.
But my interpretation is still more fun!
...given article being LLM slop their code probably also was
It would have been more precise to say O_DIRECT instead of io_uring. The point of io_uring is to avoid syscall overhead. What they were after was actually managing a page cache on their own, and it turned out to be more complicated than they thought.
The article's subheadings suggest this is AI slop. Seeing a section titles "The Production Symptom" is a Claudism in itself.
That heading is the smoking gun.
You're absolutely right.
production symptom? is this equivalent to "kidney disappointment"?
Saying that an application uses io_uring for input/output provides almost zero information. It provides about the same information as saying that an application is run on Linux.
io_uring is an alternative method of invocation for a very large number of traditional Linux syscalls, and the I/O performance that you obtain depends on which syscalls you use and how you schedule them.
You can use io_uring with classic sequential files, in order to perform the syscalls asynchronously, so you can do other processing in parallel with them, without using multiple threads.
You can use io_uring with mmap, in order to obtain much better performance than they got with mmap.
The file readahead that the kernel does when the user generates page faults is not good enough for reaching maximum performance and it has a great overhead.
For maximum performance with memory-mapped files, the user must take responsibility for the readahead, instead of expecting the kernel to do it.
This is done with madvise, but the best results are obtained by avoiding the POSIX-defined advices, which have wrong semantics, and by using 4 Linux-specific advices:
either MADV_PAGEOUT or MADV_COLD are used to inform the kernel about the pages of the file that can be freed because they will not be reused soon, so that the kernel will have available memory where to do file readahead;
either MADV_POPULATE_READ or MADV_POPULATE_WRITE are used to tell the kernel to read ahead immediately some part of the file instead of waiting for page faults, so if these are used at the right times there will never be any page faults, wasting time.
The madvise syscall must be executed through io_uring, so that it will be executed asynchronously.
When not even controlling the file readahead with madvise provides enough performance, one can replace the memory-mapped files with files opened with O_DIRECT, where all the read/write syscalls are done through io_uring.
By using fixed read/write buffers, O_DIRECT removes the overhead of mmap where each time when pages are read from the file the virtual address translation tables must also be rewritten, and the even greater overhead that happens when pages are freed and all the other processor cores must be informed about this (TLB shootdown).
However, to obtain the performance achievable with O_DIRECT, the programmer can no longer rely on the kernel for I/O scheduling, but this must be handled in the application program, which can bring significant complexity.
This means that all the file reads must be launched enough in advance, so that they will be completed by the time the application needs the data, but one cannot launch too many file reads in advance, because that would consume too much memory.
Whenever O_DIRECT is added without rewriting completely how file I/O is handled in the application, that will lead to reduced performance, not to better performance.
> The internet is full of posts declaring io_uring wins over mmap.
Internet is also full of idiots not having done their homework.
Blindly throwing io_uring for mmap and hoping for better perfomance in a highly concurrent environment is a recipe for thread contention and latency spike.
Add to that the kernel lock contention on mmap. Later kernels have tried to increasingly mitigate this. But significant latency on kernel lock contention still exists.
Try mmap + numa like pinning (from software atleast via hashing the work id) and always have the same mmap be used from the same node. This usually yields better latency compared to throwing wonder weapons.
Our database Dip uses mmap burst for opening massive amounts of mmaped kv engine files along with numa pinning of the same mmap kv engine file to the same core so as to reduce cross cache contamination and page cache relocation.
io_uring (available only in more modern kernels) has so far only increased Dip's latency. Hence we have no reason yet for using it. It is a net negative for our usecase.
Use io_uring where appropriate and don't expect every mmap replaced by io_uring to do wonders.