I haven't tested the newest implementations of every large context window model, so I'm not sure how prevalent this issue still is, but generally speaking the context window tends to be U-shaped. In other words, the model seems to forget/ignore everything in the middle. So YMMV if you're trying to implement RAG-esque methods with them.

More reading on that problem if you're curious: https://arxiv.org/pdf/2307.03172.pdf