> might have said the same thing if my mail carrier were a woman (which she is).

Probably. Most people who look like women are. But some are actually closeted non-binary people or trans men. In which case calling them a lady isn't offensive, but it hurts.