I wish there were, in C, some equivalent of "struct" but where you could specify
- The byte order / endianness
- The alignment of variables
- The exact width of variables (32-bits, 64-bits)
"struct" is great for what it was designed for, storing your internal data structures in a way efficient for the machine.
But everyone abuses structs and tries to read external data sources e.g. files using them. They might hack it to work on their own machine, then as soon as a machine of the other endianness comes along, hacks and #ifdefs appear, then machines with ints of different widths come along....
Of course these people are using structs "wrong", like the author of the article suggests. But nevertheless, the fact that people are using structs "wrong" suggests there is a need for something that provides what people are trying to use structs for.
> I wish there were, in C, some equivalent of "struct" but where you could specify
I don't think there's a need for a struct-equivalent, I think there's a need for a struct loader with these capabilities, it would only be in charge of packing and unpacking but would shove everything in a struct.
Basically, Erlang's bit syntax for C (thought the bit syntax unpack to locals, not structs):
<< Foo:16/little, Bar:12/signed, Baz:4 >> = Bin.
(default type specifications are integer, unsigned and big-endian, between the : and / is the size of the data in "units", where Unit defaults to 8 bits for the binary type and 1 bit for integers, floats and strings).
Doesn't natively do alignment though, the developer has to pad on his own.
Python's `struct` module is similar[0]: http://docs.python.org/library/struct.html although the format string is basically unreadable and I believe it's absolutely terrible at decoding non-standard sizes (e.g. an int stored on 3 bits)
[0] libpack[1] for C, Perl also has this[2] which was probably the inspiration for Python
Yeah, C is really missing a way to serialize/deserialize data from raw memory/sockets into structs usable by your code. The least insane way, libpack [1] requires replicating the data format definition three times:
* define the struct with all elements
* define a string for the binary representation
* call fpack/funpack with the string and all the struct elements as parameters...
Unfortunately, fixing this either requires some kind of black X-macro [2] magic or another template language used to write the specification and to generate the three above-mentioned representations from it...
Surely this could be handled via simple syntactic extensions to the struct specification (with everything wrapped into an ungodly macro from hell) in order to define the mapping between the struct itself and libpack's format string, no?
> The problem is that you need to replicate the struct entries in the pack/unpack calls as well
Don't you only need the (generated) format string? Ideally, the macro could generate some wrapper function of some sort as well, which would unpack, fill and return an instance of the struct.
I've been working on some code to parse the shapefile format, which specifies some fields in big-endian and some in little (I have no idea why), but the binary package has made it really easy to deal with.
> You can't tackle the BO issue here, as it is a property of the underlying architecture.
I disagree; if you have "littleendian int32 myfield" for example, every time you reference myfield on a big-endian architecture, the compiler inserts the necessary byte manipulation code, just like the guy does manually in the original post.
I agree. I think you may have misunderstood my comment. The wire representation should be read into memory via any byte order manipulations that are needed.
Why "every time you read the variable" then? You'd have a packed-binary-struct and a memory-representation-struct and conversions between them to use when reading or writing. I say "would", but I actually already do this. I have typedefs of the kind
typedef union {
uint8_t bytes[sizeof(uint64_t)];
uint64_t native; /* alignment hint */
} uint64le_t;
which I use in structs of my on-disk data structures.
It would, in the worst case, not be worse than what the original poster is suggesting.
And at least in that case, the code would be more readable.
But, as the compiler knows more about what's going on (it's not just parsing and compiling a general expression with ORs and shifts) then it could well be faster (e.g. if there were a CPU instruction to do this, then it could be used, etc.).
I'd be curious to know why you want it as a language feature rather than something like google protocol buffers. I don't necessarily disagree, I'm just curious what your reasoning is.
That's a good question. (One I hadn't thought about much before I have to admit.)
I think it would be easier for the programmer to use a language feature than a library; the resulting code would be easier to read. I'm thinking how regular expressions are easier to use in Perl than they are in Java because they're part of the language, or how maps are easier to use in scripting languages than say Map in Java because they're part of the language.
If you had to have some e.g. string or external file describing the syntax of the hardware-independent struct, and then some calls like "read_entity(void structure, char fieldname)", it would all get nasty - you're going to have strings in the code which can't be checked at compile-time, you're going to be doing type casting which can't be checked at compile-time, and so on.
But a code generation system could be an option - you define the structure in a file in a certain syntax, and then code is generated with the right types and attribute names being visible to the compiler.
But I still think being part of the language would just be simpler for the user, and I don't consider this to be some obscure feature which would dilute the purity of the language by its introduction; binary file formats and protocols are here to stay.
P.S. Yes Google Protocol Buffers could well be the thing I've been searching for since I first saw the C struct many years ago.
Yeah, protocol buffers are a bit heavyweight, a bit harder to read than a language feature could be, and add an extra dependency. That's why I don't necessarily disagree, though the reason I asked is it occurred to me that any time I have personally needed to deal with byte ordering has been code that can easily justify a heavyweight solution (unlike regex's which end up in tiny scripts that may only be run once) and they have also all been places where being able to easily interact with with the same data in other languages would have been super useful, as would being able to easily add to the format without a versioning headache.
... of course, now that I write that I realize a language feature could actually provide all of that, too.
- The byte order / endianness
- The alignment of variables
- The exact width of variables (32-bits, 64-bits)
"struct" is great for what it was designed for, storing your internal data structures in a way efficient for the machine.
But everyone abuses structs and tries to read external data sources e.g. files using them. They might hack it to work on their own machine, then as soon as a machine of the other endianness comes along, hacks and #ifdefs appear, then machines with ints of different widths come along....
Of course these people are using structs "wrong", like the author of the article suggests. But nevertheless, the fact that people are using structs "wrong" suggests there is a need for something that provides what people are trying to use structs for.