> On Linux write to a regular file never blocks. Writing to a file copies data from our user space buffer to the kernel buffer and returns immediately. At some later point in time the kernel will send it to the disk. A regular file is hence always ready for writing and epoll wouldn’t add anything.
Is that true? If it would be, the amount of data the Kernel would need to buffer is unbounded. I assumed there is a limit on the amount of buffered and not yet committed data, and when that is cross the call would block until more data is flushed to disk. Which is kind of the same as happens for TCP sockets. The `write()` call there doesn't really send data to the peer, it just submits data to the kernels send buffer, from where it will be asynchronously transmitted.
Edit: Actually I will answer my own question and say I know it will block. I had deployed IO heavy applications in the past with instrumented read/write calls for IO operations in a threadpool. Even though typical IO times are well below 1ms, under extremely high load latencies of more than 1s could be observed, which is far from "not blocking".
Yes, file I/O can block. However, there is an assumption that file I/O will never block "indefinitely" -- unless something is severely broken, the kernel will always finish the operation in finite time probably measured in milliseconds at most. The same is not true of network communications, where you may be waiting for an event that never happens.
There is a temptation to say that, well, milliseconds are a long time, so wouldn't we like to do this in a non-blocking way so we can work on other stuff in the meantime?
But... consider this: Reads and writes of memory also may block. If you really think about it, the only real difference between main memory blocking and disk blocking is the amount of time they may block. And with modern SSDs that time difference is not as large as it used to be.
So do you want to be able to access memory in a non-blocking way? Well... you can make the same logical arguments as you do with file I/O, but in practice, almost no one tries to do this. Instead, you separate work into threads, and let the CPU switch (hyper)threads whenever it needs to wait for memory.
In fact, memory reads may very well block on disk, if you use swap!
Given all this, it stops being so clear that async file I/O really makes sense.
Meanwhile, as it happens, the Linux kernel was never really designed for async file I/O in the first place. When you perform file I/O, the kernel may need to execute filesystem driver code, and it does so within the same thread that invoked the operation from userspace. That filesystem code is blocking. For the kernel to deliver true async file I/O, either all this code needs to be rewritten to be non-blocking (which would probably slow it down in most cases!), or the kernel needs to start a thread behind the scenes to perform the work.
But... you can just as easily start a thread in userspace. So... maybe just do that?
(Or, the modern answer: Use io_uring, which is explicitly designed to allow a userspace thread to request work performed on a separate kernel thread, and get notified of completion later.)
> However, there is an assumption that file I/O will never block "indefinitely" -- unless something is severely broken, the kernel will always finish the operation in finite time probably measured in milliseconds at most. The same is not true of network communications, where you may be waiting for an event that never happens.
Doesn’t this statement assume all filesystems are local? If the filesystem is remote (NFS, CIFS/SMB, WebDAV or SSH via FUSE, etc), then regular file IO is the same thing as network communication - and an NFS server going down (or losing connectivity to it) no more counts as “severely broken” than the same scenario for any other network protocol/application/service.
A network filesystem is still expected to respond to reads and writes in a short finite time, assuming the network isn't broken.
For an arbitrary network protocol, the amount of time you wait for an event is arbitrary and application-dependent. Even if everything is working exactly as intended, you could be waiting indefinitely for an event.
The network could be completely fine but the NFS server could have crashed (and maybe the backups were faulty so it is never coming back). That’s absolutely no different than if the same thing happened to an SSH server or a HTTP server or any other network service. The assumption is no more valid for NFS than it is for an arbitrary protocol.
> However, there is an assumption that file I/O will never block "indefinitely"
No, there absolutely is not.
> unless something is severely broken, the kernel will always finish the operation in finite time probably measured in milliseconds at most
Systems get busy all the time and encounter delays larger than this all the time. I have fixed many supposedly low latency systems which incorrectly (as you suggest) assume that local disk writes ought to be fast because usually it hits a disk cache. It's very, very easy to break this assumption, like by copying a moderately sized file.
You cannot assume that a disk write will finish quickly. That is not the guarantee.
> Reads and writes of memory also may block.
Not really, no. There are many orders of magnitude of difference.
> If you really think about it, the only real difference between main memory blocking and disk blocking is the amount of time they may block.
All syscalls take time, but the difference here is many orders of magnitude. If a syscall is expected to take a meaningful (or in the case of i/o, infinitely variable) amount of time we label it a blocking call.
> In fact, memory reads may very well block on disk, if you use swap!
System calls can't swap. High latency jobs will often mark themselves unswappable for exactly this reason, or systems will simply run swapless.
> Given all this, it stops being so clear that async file I/O really makes sense.
No. No no no. This is flatly wrong. You will ship broken systems if you do this and people will have to call someone who knows what they're doing to fix it.
The best you can say is that statistically it won't happen often and maybe your users (or, management) won't notice until it's not your problem anymore.
> But... you can just as easily start a thread in userspace. So... maybe just do that?
Async i/o in glibc was historically implemented as simply a separate thread, fyi (prior to io_uring - I haven't looked in a decade). This is a form of async i/o.
> Not really, no. There are many orders of magnitude of difference.
If the kernel has locked your pages so it can merge or unmerge them under transparent huge pages or if you are using swap, page faults can take a really long time. People who care about async file I/O are probably already ensuring that these circumstances don't come up though.
> If you really think about it, the only real difference between main memory blocking and disk blocking is the amount of time they may block.
This is a somewhat confusing analysis you have here. Direct read/write from memory for all intents and purposes doesn't block. Why do you say that reads and writes may also block?
The reason memory blocks is because it needs to page in or out from secondary storage - which makes this statement "the only real difference between main memory blocking and disk blocking is the amount of time they may block." not really true
> Direct read/write from memory for all intents and purposes doesn't block.
Sure it does! Main memory is much slower than cache so on a cache miss the CPU has to stop and wait for main memory to respond. The CPU may even switch to executing some other thread in the meantime (that's what hyperthreading is). But if there isn't another hyperthread ready, the CPU sits idle, wasting resources.
It's not a form of blocking implemented by the OS scheduler, but it's pretty similar conceptually.
> The reason memory blocks is because it needs to page in or out from secondary storage
Nope, that's not what I was referring to (other than in the line mentioning swap).
> Sure it does! Main memory is much slower than cache so on a cache miss the CPU has to stop and wait for main memory to respond. The CPU may even switch to executing some other thread in the meantime (that's what hyperthreading is).
Cache is a memory. And which cache, by the way? Even L1 cache on modern processors doesn't have 0 latency. And this is a rather poor way of describing hyperthreading - the CPU doesn't really "switch" - the context for the alternate process is already available and the resource stealing can occur for any kind of stall (including cache loads), not just memory. Calling this a "switch" suggesting it is like a context switch is very misleading. It's not similar conceptually.
In any event, by this definition even a mispredicted branch or a divide becomes "blocking" - which sort of tortures any meaningful definition of blocking.
The essential difference is - memory accesses to paged in memory (and branch mispredictions, cache misses) are not something you typically or reasonably trap outside of debugging. mmaps, swaps, disk I/O, network accesses are all something delegated to an OS - and at that point are orders of magnitude more expensive than even most NUMA memory accesses. I sort of see where you're coming from - but I don't think it's a useful point.
My argument is that disk I/O is more like memory I/O than it is like network I/O, and so for concurrency purposes it may make more sense to treat it like you would memory I/O (use threads) than like you would network I/O (where you'd use non-blocking APIs and event queues).
> My argument is that disk I/O is more like memory I/O than it is like network I/O
It depends on your network and disk - and yes SSD and "slow" ethernet are the common case, but there is enough variation (say an relatively sluggish embedded eMMC on one end and 100 GbE for the networking case), that there's no point in making some distinction about disk vs network latency - for a general concurrency abstraction they're both slow IO and you might as well have a common abstraction like IOCP or io_uring.
> concurrency purposes it may make more sense to treat it like you would memory I/O (use threads) than like you would network I/O (where you'd use non-blocking APIs and event queues).
No, case in point, Windows had IOCP for years such that you could use the same common abstraction for network and disk. The fact that the POSIX/UNIX world was far behind the times in getting its shit together doesn't mean much.
And why, fundamentally, can you not use blocking APIs with threads for networking?
Ah but that's not the distinction I'm making at all. This has nothing to do with the physical latency of the network, which can easily be less than that of a disk. It's about the application-level semantics of your communication.
When you are waiting for a connection to arrive on a listen socket, you have no idea how long you might wait. It might never happen. Someone elsewhere on the network has to take the initiative to connect. Depending on the protocol, the same can be true of waiting for a read or even a write.
> Windows had IOCP
I think Windows is over-engineered in this respect. It doesn't seem to have given Windows any recognized fundamental advantage in running databases.
> And why, fundamentally, can you not use blocking APIs with threads for networking?
Well, you can, especially if you're only talking to a single peer. But when you are talking to multiple parties (especially as part of the same logical task, e.g. talking to multiple back-ends to handle one incoming request) it gets unwieldy to manage a thread for each connection.
I think the important practical distinction is whether or not these stalls imply a context switch. Software that avoids blocking calls is largely trying to minimize context switches, with measurable adverse effects being increasingly common due to improvements in hardware. Stalls that do not imply context switches, such as filling a cache line, are not "blocking" as a matter of practical semantics because there is no context switch that has to be accounted for. Of course, this gets into a gray area with things like hyper-threading that have some of the side-effects of a context switch without an actual context switch.
With the utmost respect, I’ve never heard “blocking” described as “takes some measurable amount of time” (which is how I’m reading your above statement); by that definition, async blocks to a degree too.
You’re throwing traditional blocking/non-blocking distinctions on their ear.
Blocking in this case is referring to the CPU thread sitting idle whilst the operation is performed. This is what it means when your blocked on a network request, blocked on a disk operation, or blocked on a memory request. It's all blocking.
A cache miss and going to RAM is usually fast enough that we as software engineers don't care about it, and in fact our programming language of choice may not even give us a way of telling the difference between a piece of data coming from a CPU register or L1 cache vs going to RAM, but that doesn't mean the blocking isn't happening.
EDIT: to maybe make this a little clearer for those who might not be aware the CPU doesn't go fetch something from RAM. It puts in a request to the memory controller (handwaving modern architecture a bit here) then has to wait ~100-1000 CPU cycles before the controller gets back to it with the data. Depending on the circumstances the kernel may let that core sit idle, or it may do a context switch to another thread. The only difference between this process and say a network request is how many CPU cycles before you get your results. In the meantime the thread isn't progressing and is blocked.
> A cache miss and going to RAM is usually fast enough that we as software engineers don't care about it, and in fact our programming language of choice may not even give us a way of telling the difference between a piece of data coming from a CPU register or L1 cache vs going to RAM, but that doesn't mean the blocking isn't happening.
Yes, this is the line being discussed, and I guess (historically) I’ve just considered “a cost” without dragging “blocking” into the equation. We know that not accessing memory is cheaper than accessing it, and we can tune (pack our structs, mind thrashing the cache), but calling that blocking is still new to me. I’ll have to consider what it means. And also, does it imply the existence of non-blocking memory (I don’t think DMA is typically in a developers toolkit, but…)?
> And also, does it imply the existence of non-blocking memory
Prefetching instructions, to tell the processor to load before you use it!
The first google hit [0] even calls it non-blocking memory access!
In [1] you can see some of the available prefetching instructions, and in [2] some analysis on how they deal with TLB misses (another extremely expensive way memory access can be blocking short of a page fault).
Another thing not mentioned above is that accessing a page of newly allocated memory often causes a page fault, since allocation is often delayed until use of each page, for overcommitting behavior - same for writing to memory that is copy-on-write from a fork!
> And also, does it imply the existence of non-blocking memory
Yes actually! If you know your going to need a block of memory before you actually need it, you can put in a request to the memory controller before you need it, then proceed to do some other work and check back in when your ready for the data or when the memory controller signals you it's done. It's just that this kind of thing is usually the scope of compiler optimizations or hyper optimized software like Varnish cache rather than something your average web developer thinks about. It's again conceptually the same as an async network request, but you bother with one while considering the other just "a cost" because of the different timescales.
Historically I/O or peripherals were generally distinct from different types of memory.
Technically the CPU also "blocks" when executing an instruction, let's say adding two numbers, because obviously the sum isn't available in 0 cycles. One might imagine semantics where you explicitly ask some CPU unit to add some numbers and then wait for the result and having blocking or non-blocking wrappers around it but it sort of becomes moot. Generally async/non-blocking operations are abstractions over I/O peripherals that have an async nature, e.g. you submit something and you get some reply later, or you wait for a network packet to come in etc. Reply is often an interrupt.
I agree the lines have blurred a little with modern CPUs (and arguably were always blurry with some peripherals being memory mapped) but something like waiting on a packet to come in or a disk operation to complete is outside the box that you'd draw around the CPU and its more tightly integrated components.
Why do you say that reads and writes may also block?
Let's define "may block" first, perhaps? What do we mean when we say "network I/O may block"? Usually, this means that the kernel may see your network request and raise you a context switch while it waits for the network response on your behalf. In your last sentence you appear to argue that the reason why the kernel performs a context switch is relevant in determining if an operation "may block", and the GP is arguing that that's a distinction without a difference.
If the definition of "may block" is really just "the kernel may decide to context-switch away from your program", then yes, the GP's assertion that file I/O, memory I/O (mmap) and memory access (swap) are all operations that may block is correct -- the only difference is in degree: from microsecond delays for nvm-backed swap to multi-second delays for network transfers.
Or, of course, I may have misunderstood the GP's train of thought.
>> If you really think about it, the only real difference between main memory blocking and disk blocking is the amount of time they may block.
>
> This is a somewhat confusing analysis you have here. Direct read/write from memory for all intents and purposes doesn't block. Why do you say that reads and writes may also block?
Reads and writes from actual, physical, hardware memory might block, depending on how you define "block", in the sense that some reads may miss CPU cache. But once you get to that point, you could argue that every branch might block if the branch misprediction causes a pipeline stall. This is not a useful definition of "block".
The thing is, most programs are almost never low-level enough to be dealing with memory in that sense: they read and write virtual memory. And virtual memory can block for any number of reasons, including some pretty non-obvious ones like. For example:
- the system is under memory pressure and that page is no longer in RAM because it got written to a swap file
- the system is under memory pressure and that page is no longer in RAM because it was a read-only mapping from a file and could be purged
-- e.g. it's part of your executable's code
- this is your first access to a page of anonymous virtual memory and the kernel hadn't needed to allocate a physical page until now
- you're in a VM and the VMM can do whatever it wants
I think what I'm saying is that calling file I/O "blocking" is also not a useful definition of "block". Because I don't really see the fundamental difference between "we have to wait for main memory to respond" and "we have to wait for disk to respond".
> this is your first access to a page of anonymous virtual memory and the kernel hadn't needed to allocate a physical page until now
And said allocation could block on all sorts of things you might not expect. Once upon a time I helped debug a problem where memory allocation would block waiting for the XFS filesystem driver to flush dirty inodes to disk. Our system generated lots of dirty inodes, and we were seeing programs randomly hang on allocation for minutes at a time.
> I think what I'm saying is that calling file I/O "blocking" is also not a useful definition of "block". Because I don't really see the fundamental difference between "we have to wait for main memory to respond" and "we have to wait for disk to respond".
In addition to the point elsewhere made that you're sort of implicitly denying the magnitude of the differences here - the latency differences are on the order of 1000s.
The other way of separating is if the OS (or some kind of software trap handler more generally) has to get involved. A main memory read to a non-faulting address doesn't involve the OS - ie it doesn't ever block. However faulting reads, calls to "disk" IO, and networking IO (ie just I/O in general) involving the OS/monitor/what have you are all potentially blocking operations.
It does not matter whether the OS is involved. Consider a spinlock; if it is spinning, waiting on the lock to be released, then it is blocking.
What matters is whether control returns to the process before the operation is complete. If the process waits, it is blocking (aka synchronous); if the process does not wait, it is non-blocking (and possibly also asynchronous if it checks later to see if the operation succeeded).
> Because I don't really see the fundamental difference between "we have to wait for main memory to respond" and "we have to wait for disk to respond".
The difference, conservatively, is a factor of 1000.
There are plenty of times in software engineering where scaling 1000x will force you to reconsider your architecture.
To be clear I do not believe that async disk I/O is never useful, I just think that it's not as useful as people at first imagine when they learn about async I/O.
Yes, it may be 1000 times slower than memory. But there's a fundamental paradigm difference from network events, in that with network events you are waiting for some other entity to take action, with no implicit expectation that they will do so in any particular timeframe. Like, if you're waiting for connections on a listen socket, there's no telling how long you will be waiting.
Disk I/O is fundamentally different in that once you submit an operation, you expect it to complete within a reasonable, finite time period.
Async disk I/O is primarily useful for implementing read-ahead / write-behind scheduling behaviors. While databases tend to be the obvious use case, the OS is often so poor at this that there are large performance improvements even for much simpler use cases that are otherwise disk I/O intensive.
I'm not sure that's the primary use case any more. Fast SSDs require high queue depths to use their full throughput, so async IO is desirable to use any time an application knows it has several IO requests to issue in parallel—one thread per request has too much overhead.
Sure, but that behavior is effectively read-ahead / write-behind on your I/O buffers. That doesn't mean much more than anticipating future I/O operations before completion of that I/O is required by the code for efficient forward progress.
They're really not equivalent. Read-ahead only helps for predictable IO patterns. Issuing multiple read requests in parallel from the application is useful in a far broader range of scenarios. And for both reads and writes, being able to submit IO in batches (without having to wait for the entire batch to complete) can drastically cut down on overhead compared to submitting IOs sequentially as if they were a linear dependency chain, and makes it possible to keep the storage properly busy instead of it idly waiting on the host software to prepare and submit the next IO.
All cache replacement algorithms are literally equivalent to universal sequence prediction problems, per the optimality theorem. There is no implication of sequential decisions here. When you schedule a batch of disk I/O, you are essentially front-running the sequence predictor to avoid classes of prediction failure where successful prediction would be computationally intractable (and therefore not implemented in real systems), which is expected to produce better I/O throughput on average if done competently per the same theory. There is nothing magic about this, it is in the literature, and databases in particular have explicitly exploited non-sequential scheduling to circumvent fundamental sequence prediction limits for decades. Optimally anticipating future requirements for reads and writes can be called whatever you like, but that remains the primary use case for async I/O since you can't do it with blocking I/O in a single thread.
This becomes more important as caches become larger because cache efficiency increases are strongly sublinear as a function of size, as expected. Servers are already at the scale where very deep async I/O scheduling is required for consistent throughput with high storage density, beyond what can be done via traditional buffered disk I/O architectures, async or not. It is an active area of research with some interesting ideas.
The reason memory access can block is because it can cause the page fault handler to be invoked (https://www.kernel.org/doc/gorman/html/understand/understand...). There are many reasons the page fault handler might cause the process to block. The kernel made need to swap the page in from disk, or it might be a copy-on-write page that was just written to.
To copy the page, the kernel needs to first find a free page frame. If there are no free page frames, the kernel will attempt to reclaim pages that are in use (https://www.kernel.org/doc/gorman/html/understand/understand...). This may cause in process-mapped pages being swapped to disk, but it also may result in disk writeback activity (https://lwn.net/Articles/396561/). In either case, control cannot be returned to the process until there is a free page to map into the process.
It's complicated, memory accesses can really block for relatively long periods of time.
Consider that regular memory access via cache takes around 1 nanosecond.
If the data is not in top-level cache, then we're looking at roughly 10 nanoseconds access latency.
If the data is not in cache at all, we are looking into 50-150 nanoseconds access latency.
If the data is in memory, but that memory is attached to another CPU socket, it's even more latency.
Finally, if the data access is via atomic instruction and there are many other CPUs accessing the same memory location, then the latency can be as high as 3000 nanoseconds.
It's not very hard to find NVMe attached storage that has latencies of tens of microseconds, which is not very far off memory access speeds.
I just want to add to your explanation, that even in the absence of hard paging from disk, you can have soft page faults where the kernel modifies the page table entries or assigns a memory page, or copies a copy on write page, etc.
In addition to the cache misses you mention there's also TLB misses.
Memory is not actually random access, locality matters a lot. SSDs reads, on the other hand, are much closer to random access, but much more expensive.
io_uring just racked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it. The idea that you can just hand off infinite amounts of work for the kernel to do on your behalf is pretty fundamentally broken. It is a concrete implementation of wishful thinking.
That analysis would seem smart but let's try a game of Mad Libs:
The Linux Kernel just racked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it.
KDE just racked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it.
Firefox just racked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it.
Chrome just racked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it.
Windows just racked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it.
Photoshop just racked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it.
All CPUs just rucked up another CVE, so I kinda feel that its severely under-designed nature will always haunt it.
What's the theme? Racking up CVEs is something all software & hardware does. Mistakes can happen in design and in implementation and no one is immune. Using presence of CVEs as an indication of immaturity / fundamental design flaw isn't helpful. In fact, it's probably the opposite. Software that has no CVEs probably just means no one is paying attention to it. Sure, in a theoretical case maybe you've built a formal proof and translated that into a memory safe language somehow (& you assume you've made no mistakes modelling your entire system in your proof), then maybe. However, that encompasses 0% of all software.
> The idea that you can just hand off infinite amounts of work for the kernel to do on your behalf is pretty fundamentally broken. It is a concrete implementation of wishful thinking
How is that any different from a file descriptor? The kernel is free to setup limits on how much work you can have outstanding at any given time (now maybe those bits are missing right now, but it doesn't feel like an intractable problem).
All "work" you want to do that interfaces with anything on an OS is handed off to the kernel; want to read a file? kernel, want to sleep for a while? kernel, etc. Besides things like network traffic is also asynchronous like io_uring (even if the socket() interfaces make it look somewhat synchronous). Outside of toy system asynchronicity is always a thing, especially when running on multiple cores.
I kind of get where you are coming from but at the same time, the kernel always gets the last say, so as long as io_uring has a good design and implementation it will always be just as good or bad as the OS as a whole. Whether run of the mill programmers are up to the task of being able to properly conceptualise and use such an OS is probably not the same thing.
Yeah but it's not well-designed, that's my point. It has obliviously shrugged off the tricky question of object lifetime, that's why it has already collected 16 different CVEs for things like use-after-free. Considering its short history, io_uring has already rocketed to the top of the list of dangerous kernel features.
my understanding is that this hook is the only one, but to properly secure io_uring one has to implement a check for every call here. or just disable it.
The term "blocking" in UNIX-like OSes is jargon with a particular meaning. It means an interruptible wait.
Disk files do not block - they may Disk Wait instead, which is an uninterruptible wait (this is what the 'D' process status stands for). Disk Wait doesn't interact with O_NONBLOCK, select(3), poll(3) etc.
(Back in the bad old days it wasn't even possible for a Disk Waiting process to wake up to process a SIGKILL and die, which was the bane of system administrators everywhere when NFS introduced the idea of disks that could disappear when the network went down. Now it's common for OSes to make some kinds of Disk Waits at least killable).
> I assumed there is a limit on the amount of buffered and not yet committed data, and when that is cross the call would block until more data is flushed to disk.
There is. It's tunable through /proc/sys/vm/dirty_ratio. When there is that much write cache, application writes will start to writeback synchronously.
There is also dirty_background_ratio, which is the threshold at which writeback starts happening in the background (that is, in a kernel thread).
> The physical memory is volatile and the common case for getting data into the memory is to read it from files. Whenever a file is read, the data is put into the page cache to avoid expensive disk access on the subsequent reads. Similarly, when one writes to a file, the data is placed in the page cache and eventually gets into the backing storage device. The written pages are marked as dirty and when Linux decides to reuse them for other purposes, it makes sure to synchronize the file contents on the device with the updated data.
There are many advantages to doing it this way. One is that multiple writes to the same page will result in a single physical write, if the page has not yet been flushed to disk.
There are many reasons that you might have seen a write to a file block. One is that the number of dirty pages has reached the threshold (nr_dirty_threshold in /proc/vmstat). After that happens, any process doing disk IO will block.
Another reason is memory pressure. Since all writes go through the page cache, the kernel must first allocate a page before the call to write(2) can be completed. If there are many pages in the page cache, this can take a long time (I once witnessed an old kernel bug cause all page allocations to result in kswapd attempting to reclaim pages, due to active pages being placed ahead of inactive pages in the LRU lists).
In general, if you are writing a lot to disk but you have no intention of reading it in the near future, it is a good idea to call posix_fadvise(2) with FADV_DONTNEED to ensure the pages will be reused for something else more quickly.
It is pretty easy to completely hork a large box with a very disk intensive process; hit a local file system hard enough and you can get a majority of the processes into D state, uninterruptible IO Disk wait. Maybe not from inside a container, haven’t see it, but definitely on a box with shared processes. Even just too much logging can harm unrelated processes that aren’t even doing much with the disk.
> The number of bytes written may be less than count if, for
example, there is insufficient space on the underlying physical
medium, or the RLIMIT_FSIZE resource limit is encountered (see
setrlimit(2)), or the call was interrupted by a signal handler
after having written less than count bytes. (See also pipe(7).)
That doesn't answer the question. Blocking isn't a matter of how much data is written, but a matter of when the system call completes. Other parts of that man page imply that write(2) may block, unless the fd was opened with O_NONBLOCK (in which case you'll get an EAGAIN error instead of it blocking).
"It's complicated." Generally, with a regular file, write(2) will complete as soon as the the data makes it to filesystem buffers/cache. The data is probably not on disk when the call completes. This depends on how the file was opened (O_FSYNC, O_DIRECT, etc.) and the underlying filesystem itself. There are many other details at work, like actual file system, memory pressure (there may not be enough buffers), cache in the physical disk device or controller, etc. So the write call itself is "blocking", but the physical writes are (generally) not synchronous with the call.
Yes, whether a write blocks is really about whether the application can do anything else while the write is processed; whether the application is told the write is done when it lands in a cache or when it is actually on stable storage is a separate question.
I don't know if I'm contributing anything, but to clarify by going back to basics (rather than advanced or non standard system calls),
all writes (and reads) are individually atomic, so if you and I are writing to the same file at the same time (in real time), how our overwriting behaves is relatively well behaved and can be relied on, so there is at least that level of atomic blocking, and that's important because
the kernel also buffers files transparently, in that our two processes will share the same buffers, so beyond atomicity in a shared piece of the file, we won't get blocked wrt each other. But your reads need to pick up my writes (atomicity!) so there isn't going to be buffered queuing of writes on top of the disk buffering.
so if the kernel decides it needs to spend more time syncing dirty buffers to disk, it's not blocking our processes in the regard above, but the system could become net i/o bound. This is 1/2 the reason to make sure to use virtual memory, so plenty of a flexible amount of disk buffer will be available (the other half being to swap out resident but no longer referenced dirty pages, as from the initialization code in your app)
No, as you reasoned out it is absolutely incorrect. Write calls to regular files will block until they are complete, unless some kind of error situation is encountered.
This effect is often particularly pronounced with NFS, where calls might block for hours or even indefinitely if the underlying network filesystem goes away.
> This effect is often particularly pronounced with NFS, where calls might block for hours or even indefinitely if the underlying network filesystem goes away.
I can't tell you how many times I've had to debug a stuck process and it turns out that the logs indicated the NFS had a hiccup a day or two ago during a file read or write and the process was never notified of a file error. It's f!@#ing frustrating. Worse, though, was CIFS.
I routinely have to run file system scans on a giant NFS filer, and even without a hiccup, out of a 100M stat or read calls, ten or so will just never finish. In Go, I have to wrap the call with a channel thing and a time out and hope I don’t run out of threads before scanning all 400 M files.
Just in case anyone isn't aware there is a mount flag called "soft" that allows the NFS client (and some other network filesystems) to timeout or be interrupted, i.e. the process won't get stuck in 'D' (device wait) state.
No, that is incorrect. I think you're confused as to what "blocking syscall" means.
A blocking syscall doesn't always block, but it can always block because its interface is designed to block. Write(2) to a disk file can always block, 100% of the time. It is always considered a blocking syscall, even when it happens to not take very long due to cache optimizations.
Every single blocking syscall can sometimes return right away. This has no bearing on whether it is, or isn't, a potentially blocking call.
GGP was objecting to the fact that write always return immediately (writing to memory buffer, no back pressure), and GP said:
> Write calls to regular files will block until they are complete
I therefore understood "complete" to be the opposite to "only into buffers" in that context, e.g. all the way to disk. That is what I'm objecting to.
As to write(2) being a "blocking syscall" in the technical sense, I don't think I agree either. Of course it has that potential, but a write(2) to a regular file does not block in the sense of an interruptible block, or in the sense that O_NONBLOCK or select() can be used to avoid it. It is a "disk sleep" that is generally very different.
Blocking calls often (perhaps usually) don't block. This has nothing to do with the definition.
> As to write(2) being a "blocking syscall" in the technical sense, I don't think I agree either.
Well, you are wrong.
> Of course it has that potential, but a write(2) to a regular file does not block in the sense of an interruptible block, or in the sense that O_NONBLOCK or select() can be used to avoid it.
These have nothing whatsoever to do with whether a syscall is blocking or not. You can't use them on wait(2) either, can you?
Please stop arguing about things you clearly know nothing about.
> Please stop arguing about things you clearly know nothing about.
You could have shown your true color outright and saved me the time. You don't care what I mean, you don't care what anybody means, you will just argue semantics to find a possible meaning where you're right. Well congrats.
Sure, I will stop argumenting because you told me to. Thanks for the advice given while "you didn't care what I meant", it's as invaluable as you think.
Is that true? If it would be, the amount of data the Kernel would need to buffer is unbounded. I assumed there is a limit on the amount of buffered and not yet committed data, and when that is cross the call would block until more data is flushed to disk. Which is kind of the same as happens for TCP sockets. The `write()` call there doesn't really send data to the peer, it just submits data to the kernels send buffer, from where it will be asynchronously transmitted.
Edit: Actually I will answer my own question and say I know it will block. I had deployed IO heavy applications in the past with instrumented read/write calls for IO operations in a threadpool. Even though typical IO times are well below 1ms, under extremely high load latencies of more than 1s could be observed, which is far from "not blocking".