The main issue I have with not using swap in modern Linux is that it will cause the kernel to be busy for hours at a time. What happens is, as the kernel runs low on RAM, it has to spend more time searching for smaller and smaller chunks of RAM to back the request, the smaller chunks are more numerous and the "kswapd" kernel thread is responsible for this activity. As the system approaches 0 RAM free kswapd will also try to release less important pages, which takes more CPU time. Ultimately you get to the point where allocations take a really long time, and there are lots of allocations.