Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Using the x86-64 C struct layout as the serialization format is increasingly common in financial trading protocols. Skipping all the bit twiddling has a huge performance impact at high message rates. I once compared two feeds covering the same basic data, where the major difference was one had a complicated serialization and one was C struct layout. The former needed two 16 core servers just to process I/O and deserialize the messages; the latter used 1 core on a single server.


"The former needed two 16 core servers just to process I/O and deserialize the messages; the latter used 1 core on a single server."

I'm genuinely curious; what was the 'former' protocol? Was it encoded using FAST or zlib or something?


I would guess FIX/FAST. For a concrete example, have a read of the specifications for Eurex's market data protocols:

http://www.eurexchange.com/exchange-en/technology/t7/system-...

Specifically, compare the Enhanced Market Data Interface protocol (described in "T7 Market and Reference Data Interfaces") and the Enhanced Order Book Interface protocol. EMDI is FAST, a morass of tags, stop bits, presence bitmaps, and who knows what else (i don't). EOBI is structs.


I was consuming the feed in question via an API provided by the vendor and didn't have direct knowledge of the wire protocol, but based on how it was described to me during the sales process it was probably either FAST or a proprietary protocol with a similar design.


Regarding bring lazy on the serialization... yeah packed structures make sense. But excess padding is still wasteful, and endianness is only a bswap away, which is 1-2 cycles apiece and so won't exactly break the latency budget (per Agner Fog's wonderful pdf, looking at Haswell).


It might only take a cycle to swab, but the big hit is probably the data dependency you introduce.


Which also happens to be why we used binary document formats first then moved to XML too.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: