ZFS ARC unfortunately does not integrate with the kernel file cache, so they step on each other a lot. ZFS does watch available system RAM and try to dynamically reduce its usage as memory pressure increases, but I've found its responsiveness for this to be far too slow. This combined with how ARC appears to just be an opaque block of RAM that cannot be reclaimed, I usually just set a hard limit on how big the ARC is allowed to get in the module load arguments and be done with it (at least for systems that are doing more than just storage).
At least on FreeBSD, there is a kmem_cache_reap() that is called from the core kernel VM system's low memory handlers.
Looking at the linux code in openzfs, it looks like there is an "spl_kmem_cache_reap_now()" function. Maybe the problem is the kernel dev's anti-ZFS stance, and it can't be hooked into the right place (eg, the kernel's VM low memory handling code)?
