A long time ago, I started working on something similar. There are some binary editors that support writing grammars. I worked on reverse engineering several binary file formats. When working on Quark Express file format, I discovered you do need two layers of grammars, as the file contains a header followed by allocated blocks, where the bocks are linked (by indices) and chains of linked blocks contain streams of data. I remember that there are also file formats, that simply have pointers to certain sections of the data contained in the file. I once implemented a memory mapped file data storage where all pointers to allocated blocks where represented by an offset from the start of the file. (I later, came up with a better idea and that is to make them relative to the location of the pointer, such that you do need to know the offset at which the memory mapped file is placed in memory. So basically, there are lots of 'pointers' in the file and you can only read the data in the file following those pointers.)
Lord. I still have whole books I laid out in QuarkXPress for commercial printing that I can't open. Reverse engineering the file format was not something that had occurred to me.
I have made the code available on GitHub, The code was developed on Windows and makes use of some Windows specific functions. But I think that dependency can be removed.
Unity asset files killed my attempt at a comprehensive binary encoding format spec language. There’s a header containing a list of block pointers. The blocks are compressed and the (several) files are sequential. Typically the first file is a resource containing a block of metadata describing object type formats, then object references keyed to the specified formats. The objects can have transparent references (PPtrs) to other objects in the resource file, or other files in the asset, or other files or objects in in other asset files. One asset file usually contains the master list of what object is in what file.
None of this is a true blocker, but your format has to contain sigma types and at that point you’re just writing a parser normally.
Links: https://www.iwriteiam.nl/Ha_BFF.html https://www.iwriteiam.nl/Ha_HTCABFF.html https://www.iwriteiam.nl/D0205.html#13MMF