As long as we use the traditional network protocols and socket API it's not "dead" dead. The other name for big-endian is "network order", after all.
As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing.
In one instance I was reading a spec for an industry-specific protocol. At first I was looking at it and thinking "What the... they are padding stuff in a strange way, and using little-endian for the data". Then it suddenly dawned on me, that they've designed the spec to be whatever GCC on x86-64 Linux machine would do to layout C the structures in memory.
So someone very lazy could just define a struct and cast into it as data comes from the wire. Someone one a big endian machine, would have to do a lot more legwork to get the thing working. But given that there aren't many of those around, it was deemed an acceptable tradeoff.
Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deserialize the messages; the latter used 1 core on a single server.
Specifically, compare the Enhanced Market Data Interface protocol (described in "T7 Market and Reference Data Interfaces") and the Enhanced Order Book Interface protocol. EMDI is FAST, a morass of tags, stop bits, presence bitmaps, and who knows what else (i don't). EOBI is structs.
I was consuming the feed in question via an API provided by the vendor and didn't have direct knowledge of the wire protocol, but based on how it was described to me during the sales process it was probably either FAST or a proprietary protocol with a similar design.
Regarding bring lazy on the serialization... yeah packed structures make sense. But excess padding is still wasteful, and endianness is only a bswap away, which is 1-2 cycles apiece and so won't exactly break the latency budget (per Agner Fog's wonderful pdf, looking at Haswell).
> they've designed the spec to be whatever GCC on x86-64 Linux machine would do to layout C the structures in memory.
In a previous job, the software I worked on had a "file format" that just consisted of dumping a raw array of structs to disk. It was obviously pretty fast, but it made cringe a lot.
What's worse, the compiler had flags to control if and how much struct padding to use, so reading a data file with a binary compiled with different padding from the one that created that file caused it to crash. Fun times...
The ideal is probably no padding --- on x86, unaligned accesses basically need no extra cycles[1] and if it means structures shrink and reduce cache misses, could actually be better.
[1] Unless you happen to access a field spanning two cache lines and miss, in which case the padded version would require accessing that second cacheilne anyway.
Unfortunately, the default for the compiler we used[1] to align struct members on DWORD (32-bit) boundaries, for some technical reason I am sure was totally reasonable. And these guys all built their code with a special make-variable that appended the "don't-pad-structs-I-really-know-what-I-am-doing" flag to the compiler's command line.
I did read somewhere, that x86 (at least Pentium III and later) like to make memory reads from addresses that are a multiple of four. But the profiling data I was able to gather showed that that part of the application had a negligible impact on overall performance. Since the vague job description I had gotten said my job was to "make things faster", I decided not to look into this any further.
The same was done "back in the day". On our systems, we had a reliable transport protocol (precursor to TCP/IP) that took advantage of the fact that the network and hardware was all BE. Everyone tried to squeeze every last bit of performance out of the machines and avoiding the byte shuffling was not an insignificant performance gain. It's not surprising that people are now doing the same thing with x86-64 given its dominance.
I trained myself to stop doing that (casting struct pointers over network data) after porting C code to an Alpha and getting bus errors from the unaligned accesses. And now I have to untrain myself?
> Someone one a big endian machine, would have to do a lot more legwork to get the thing working. But given that there aren't many of those around, it was deemed an acceptable tradeoff.
Even PowerPC64 has a little-endian mode - PPC64LE.
The machines are switching over because the expensive part is turning out to be software which is already working & the costs associated with migrating it over to a different endian system.
> The other name for big-endian is "network order", after all.
I wonder where this started... I always suspected it was Sun with their "The Network is the Computer" motto, and of course they would define "network byte order" as what they used.
Today, it drives me crazy that we're constantly swapping before sending across the network and then swapping again when it's received.
As way to serialize data (wire / disk format) it's becoming more common. FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely eliminate?) byte shuffling when de-serializing.
In one instance I was reading a spec for an industry-specific protocol. At first I was looking at it and thinking "What the... they are padding stuff in a strange way, and using little-endian for the data". Then it suddenly dawned on me, that they've designed the spec to be whatever GCC on x86-64 Linux machine would do to layout C the structures in memory.
So someone very lazy could just define a struct and cast into it as data comes from the wire. Someone one a big endian machine, would have to do a lot more legwork to get the thing working. But given that there aren't many of those around, it was deemed an acceptable tradeoff.