The evidence says otherwise. Scalia voted to strike down laws or held executive actions unconstitutional many times over the course of his career. At some point "the rarest of circumstances" has to have some bite. I'm sure he would have argued that he was compelled to do so by the text of the constitution and by the flagrant unconstitutionality of those laws/actions, but I don't think that's a good answer. He may have wished that circumstances would have allowed him to be an avatar of judicial restraint but regardless that's not what happened.