I've run a lot of systems with gobs of ram and like 512M swap on FreeBSD. Linux is different than FreeBSD of course, but determining how much memory is being used / if you're running out is difficult. Watching swap stats is a better gauge.
If your swap i/o rate is high, you're probably thrashing, even if the usage is low; you're right on the edge, and need more ram (or fix bugs).
If your swap usage is growing rapidly, you've got an urgent problem; sometimes a little bit of swap lets you get in and fix something or at least shut it down nicer than OOM killer, sometimes it just gives you a more obvious record that you ran out of memory.
If your swap usage gradually climbs to say 50% over time, you might have a slow leak, or something to look at anyway.
If your swap i/o rate is high, you're probably thrashing, even if the usage is low; you're right on the edge, and need more ram (or fix bugs).
If your swap usage is growing rapidly, you've got an urgent problem; sometimes a little bit of swap lets you get in and fix something or at least shut it down nicer than OOM killer, sometimes it just gives you a more obvious record that you ran out of memory.
If your swap usage gradually climbs to say 50% over time, you might have a slow leak, or something to look at anyway.