Any compressor actually can be used, trivially, as an autoregressive language model.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.
This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.
For example, LLM pretraining datasets are on the order of tens or hundreds of terabytes. If an LLM-based code for that data is ~twice as efficient as gzip, you could afford to transmit the weights of even a very large LLM and still come out ahead.
In other words: LLMs actually are excellent compressors of their training sets in the formal information-theoretic sense.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.