This! I've started to look for usable vector image format for game development somewhere in 2004, now it's 2017 and there is still nothing available. For SVG you literally need to pull-in whole browser engine, which is most likely 2-3 times the size of your whole game.
One possible solution is to the same what XMPP did on XML, or JSON did on JavaScript: Define a "safe subset" of SVG, and stick to that. Perhaps use the "safe subset of XML" as a base (as defined e.g. in XMPP.)
Use a library (or at least a validator) that works exactly on this subset, and fails for anything not strictly conforming to this.
Of course, this is not as satisfying as a "clean slate" approach, but on the other hand, this is a clear migration path with backwards compatibility - which is always good to have during a transition period to avoid all those bootstrapping and chicken-egg issues.
I've found librsvg to be pretty good. It's used for exactly this purpose in Gnome. You stick a Cairo surface behind it and it translates the operations to that.
However, admittedly, I use it in combination with a full blown browser because it doesn't need/handle text completely. I like SVG, but I too have accepted that if I want stuff to render properly, the only place to find a usable and complete SVG engine is in a web browser.