Unsafe blocks don't mean it's unsafe: just that the language's safety scheme can't prove anything about that particular block. You prove each safe externally with Rust's safety handling everything else (i.e. majority of the code).

You can also go further like I did here:

https://news.ycombinator.com/item?id=21840431

You're still worrying about a smaller portion of your code.