That's not actually true; it's an asymmetric problem. You're asking about impacting another orbiting object with nonzero relative velocity. That's not the symmetric question of transferring between two orbits A and B, both ways; it's the asymmetric question of going from orbit A to "some orbit that intercepts B at some point".

It's much less delta-v to go from the Oort cloud to Earth—to a highly-eccentric orbit that intercepts the Earth's orbit, without matching its velocity.