It's true that you'd have a hard time knowing a chip wasn't swapped someone on its way to you. There are a few things that would help but nothing easy. Getting the hardware into retail locations so that anyone could walk in off the street and buy a system with cash. Random spot checks, chip by chip. Packing the system in colored beads/rice to prevent tampering after it's in a box.

At this point, even if I could never be sure my system was secure, just knowing that systems were purchased, tested, and verified would make me feel a lot better. It'd be nice to have confirmation that it's even possible to make and sell a system that isn't backdoored in this country.