"After all, prehistoric men weren't known for their long life expectancy."

After surviving early years, people still used to get old. Infant mortality was just way higher bringing the average down. (And that those metrics often compare to poor peasants and our paleo ancestors were not peasants)