"Unfortunately, this only works because the abstraction handles it well. You can't really do custom things with file descriptors, so the amount of useful things you can do by treating sockets as files is quite limited"

In general, most code has a clear initialization step where it sets up everything it wants to set up, then you can pass it to something that only expects a io.Reader/io.Writer and it can operate. I have a couple of places where one way of getting something does an HTTP request, but another way opens an S3 file for writing, and yet another way opens a local disk file. Each of them has their own completely peculiar associated configuration and errors to deal with, but once I've got the stream cleanly I pass of an io.WriteCloser to the target "payload" function.

If you're doing super intense socket stuff you may need to grow the interface, or even just plain code against sockets directly. But most application-level stuff, even complicated stuff like "Maybe I'm submitting a form and maybe I'm writing to S3 and maybe I'm writing to disk and maybe I'm writing to a multiplexed socket and maybe I'm doing more than one of these things at once" can be cleanly broken into a "initialize the io.Reader/io.Writer, however complicated it may be" phase and a "use the io.Reader/io.Writer in another function that doesn't have to worry about the other details" phase. It is also highly advantageous to be able to pass a memory buffer to the latter function to test it without having to also try to figure out how to fake up a socket or a file or whatever.

People don't write applications that accept either files or sockets because in most languages there is one impediment or another to the process; a system that almost makes file-like objects share an interface but in practice not really, libraries that force you to pass them strings rather than file-like objects, etc. While it isn't attributable to Go qua Go, by getting it right in the standard library early Go legitimately is really good at this sort of thing, more because the standard library set the tone for the rest of the ecosystem than because of any unique language features. I hear Rust is good too, which I can easily believe. Every time I try to use Python to do this, I'm just saddened; it ought to work, it ought to be easy, but something always goes wrong.

The way I see it, Go did two things that made it work well:

- Have a very simple, fairly well-defined interface for arbitrary read/write streams. This interface needs to have decent characteristics for performance, some kind of error handling, and a way to deal with blocking. Go's interface satisfies all three.

- Have a good story for async in general. It's not really helpful to have an answer for how to deal with I/O blocking if the answer is really crappy and nobody actually wants to use it. A lot of older async I/O solutions felt very much in this camp.

I think that Rust does a pretty decent job, though I'm a little bearish on their approach to async. (Not that I have any better ideas; to the contrary, I'm pretty convinced that Rust async is necessarily a mess and there's not much that can be done to make it significantly better.)

But I think you can actually do a decent job of this even in traditional C++, in the right conditions. In fact, I used QIODevice much the way one would use io.Reader/io.Writer in Go. It was a little more cumbersome, but the theory is much the same. So I think it's not necessarily that Go did something especially well, I think the truth is actually sadder: I think most programming languages and their standard libraries just have a terrible story around I/O and asynchronicity. I do think that the state of the art has gotten better here and that future programming languages are unlikely to not attempt to solve this problem. So at least there's that.

The truth is that input and output is unreliable, limited and latent by the nature of it. You can ignore it for disk because it's relatively fast. But at the end of the day, the bytes you're writing to disk need to go through the user/kernel boundary, possibly a couple times, to the filesystem, most likely asynchronously out of the CPU to the I/O controller, to the disk which likely buffers it, and then finally from the disk's buffers to its platters or cells or what have you. That's a lot of stuff going on.

I think it's fair to say that "input and output" in this context means "anything that goes out of the processor." For example, storing data in a register would certainly not be I/O. Memory/RAM is generally excluded from I/O, because it's treated as a critical extension of the CPU and sometimes packaged with it anyway; it's fair for your application (and operating system) to crash violently if someone unplugs a stick of RAM.

But that reality is not extended almost anywhere else. USB flash drives can be yanked out of the port at any time, and that's just how it goes; all buffers are going to get dropped and the state of the flash drive is just whatever it was when it was yanked, roughly. USB flash drives are not a special case. Hell, you can obviously hotplug ordinary SSDs and HDDs, too, even if you wouldn't typically do so in a home computer.

So is disk I/O seriously that different from network I/O? It's "unreliable" (relative to registers or RAM). It's "slow" (relative to registers or RAM). It has "latency" (relative to registers or RAM). The difference seems to be the degree of unreliability and the degree of latency, but still. Should you treat a `write` call to disk differently than a `write` call to the network? I argue not very much.

I don't really know 100% why the situation is bad with Python, but I can only say that I don't really think it should've been. Of course, hindsight is 20:20. It's probably a lot more complicated than I think.

"I think the truth is actually sadder: I think most programming languages and their standard libraries just have a terrible story around I/O and asynchronicity."

Whenever I post this sort of claim, I try to make it clear that it's not really "Go triumphalism", because I agree with you. It ought to be just as easy in a lot of other languages too. It's not a matter of features, or missing features, or features at all.

C and C++ both have a number of abstractions I've seen on this idea, but they aren't compatible and not universal, so using them is a pain because you pretty much have to adapt everything yourself into whatever you are using. (C is particularly problematic; two libraries or even just two adaptors to some particular IO facility can be API compatible but still not work together properly if they have different ideas about memory ownership. C++ can still get into that problem though my perception is there's a better understanding of the problems, if nothing else. Rust has a huge advantage on that front.) Go had enough leadership that everybody has the same abstraction out of the box, and you end up very encouraged by the community to conform to it unless you have good reasons. Good reasons exist; I've got some things that wrap io.Readers but just can't be io.Readers on their own because the abstraction doesn't fit. But they are the exceptions, and I don't see them often.

> (I was going to say "TLS for example" though come to think of it this isn't even strictly true under Linux; https://docs.kernel.org/networking/tls.html )

TLS isn't that much harder to handle than regular TCP sockets--you're going to need a socket interface that lets you get reader and writer streams, and the extra roundtrips you need for negotiation are handled in the constructor for the TLS socket.

It is more difficult if you want to support more advanced features of TLS, or especially if you want to support something like STARTTLS (negotiate TLS on an already-open socket). But this is already kind of true for sockets in general: the reader/writer abstraction breaks down relatively quickly if you need to do anything smarter than occasionally-flushed streams.

I think my original post was fairly unclear of what I meant.

See, what I meant was like this. File descriptors are an OS abstraction; the "backends" you get are defined in the kernel. You can't really do custom behavior with FDs; for example, in Go, the Go TLS library can open a connection and return an io.Writer, and when you write, it will be symmetrically encrypted, transparent to you, as if the code spoke TLS. But when you're dealing with raw file descriptors, and the read and write syscalls, there's no way to make 'custom' read and write handlers, like you can with programming language abstractions.

(I do acknowledge that you could in fact do some of this with pipes, though I have seldom seen it used this way outside of shell programming. It's kinda missing the point, anyway, since pipes are just another type of fd. You can do pipes on top of Go's abstraction too, but it would be very cumbersome in comparison.)

But as a kind of quirk, Linux actually _does_ support TLS sockets. It will not do the handshake, so you still have to do that in userspace. But if you use the Linux kernel's TLS socket support, it will in fact give you an FD that you can read from and write to directly with transparency, as if it was any other file or socket; you don't have to handle the symmetric cryptography bits or use a separate interface. I think this is rather neat, although I'm not sure how practically useful it is. (Presumably it would be useful for hardware offloading, at least.)