I am only going off my memory, so I could be mistaken. But IIRC the OG pi used processors originally designed for phones (at least the one that hit the market, eraly prototypes were based on Atmel micros), The iPhone used a processor originally designed for a set-top box from Samsung which was then underclocked to save on battery.
IIRC they realised that the micros were not going to cut it, they went to Broadcom (Which Eben was working for at the time) and they were able to supply some "overstock processors" for cheap, which became the processor used in the Pi. Remember at the time the Pi was never designed to be for "makers" but to be a cheap computer to help better kickstart education, it was never designed for "us", but we all said "hey, cheap little linux computer, I'll take 5!
It could be the both things are true, it's common enough for chips to be made to serve two markets to save capital cost
I've come across some (unknown provenance) information that bcm2763, which was advertised as a phone chip on an old version of the broadcom website (via archive. org) was the same die as bcm2835, but with dram hooked up in a different way.
It's true that videocore was intended to be a GPU for phones, though.