SMCs are not the problem. The problem is code in OSX could put your key there in a way that someone could dump it.
Of course, since FileVault is not open source, we have no way of knowing if it does this. Is this paranoid? Perhaps, but if you are worried about cold boot attacks you should be worried about this as well.
You might also be worried about some strange design decisions in FileVault such as the fact that it uses public key cryptography[0] for what ought to just be symmetric disk encryption. While not a red flag,it is a bit strange.
