You're better off not storing your passphrase in the keychain and just typing it in that one first time, though. But it's still marginally better than an unencrypted private key for automation if you need a 100% unattended setup.
"In the keychain" doesn't necessarily mean in the login keychain. You can create as many user keychains as you want, with different passwords. Also, add the keychain app to your menubar (via preferences) to get a quick way to manually lock one or all keychains (and alternatively, the whole computer).
