I've wondered about stuff we leave off planet too. Like the stuff we left on the Moon. Our rovers on Mars. The JWST. Seems not entirely crazy to be looking in those places for evidence of past Earth-borne civilization.
I get that feeling hearing him debate. His recent interview with Sam Harris made him look rather foolish. He may be smart but he's not great at getting that across to an audience.