Oh wow, so it worked pretty well on data it hasn't seen. That expected but cool to reproduce.
Have you seen this leaderboard of sorts[1], and this proposal to change hutter prize[2]?
I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric because file sizes are very concrete. They are already beating shannon's numbers using a human prediction for compression, from what i can see.
>I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric
I think it's limited to using base models (non post-trained), because the post-training would skew the logit distribution. There are ways to "coax" post-trained models back into behaving "like" a base model, I wonder if the benchmark could be unofficially updated with those somehow.
Have you seen this leaderboard of sorts[1], and this proposal to change hutter prize[2]?
I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric because file sizes are very concrete. They are already beating shannon's numbers using a human prediction for compression, from what i can see.
https://github.com/hkust-nlp/llm-compression-intelligence
https://gwern.net/hutter-prize