That would work reasonably well. With one important caveat: memory controller tracks all memory, including page cache. Setting memory.min will prevent reclaim of page cache charged to that cgroup, as long as total memory usage stays below the limit.
Another important caveat: people keep assuming that page cache can be easily discarded and it doesn't count towards the working memory set. Then they get surprised by unexpected OOMs. There is a decade-old issue on Kubernetes https://github.com/kubernetes/kubernetes/issues/43916
