I remember when large-scale multi-core MIPS was the impending next generation of network processing technology; at the time, architectures specialized to packet processing seemed promising, but they seem less so now. What's the current state of play with X64 vs. custom MIPS?
Cavium was not yet sharing eval parts when I left this field, just a bit before Intel cancelled their NXP line (which was fun to play with).
Intel is closing the gap with their DPDK, but Cavium creams them on clock-by-clock and on cost of goods.
Cavium Octeon chips currently scale to 32 cores at 1.4Ghz, but with ZIP, GZIP, AES, SHA1, etc coprocessors running at 800Mhz. All cores share a fast, coherent unified L2.
One of the key advantages of the Octeon architecture is their hardware work scheduling unit. This is essentially a highly programmable hash engine on packet fields (with software-only bits for software classify-then-reschedule). The idea is to ensure that no packets with identical hashes are in flight on any core at the same time.
If programmed correctly, this work scheduling prevents data structure contention, which is particularly problematic when you scale to 32 (and next-gen up to 48 [then I believe to 64] cores).
The chips also support direct packet transport (XAUI, SGMII, etc), rather than requiring transport across PCI-e. Each of these ports can be programmed separately, so you can use switch-specific goofy encapsulation modes (Broadcom HiGig2, Marvell DSA, etc) to support very quick traffic <-> physical port mappings.
I should also mention that Cavium scales down very well, all of the way to configurations like 2 cores at 400Mhz for PoS, SOHO usage, and such. So it can be an attractive architecture to target.
Finally, Octeon family MIPS64 has a lot of MIPS64 extensions, like branch on bit, posted atomic operations (e.g. statistics, where you don't care about the value, you just want to += 42 it), pop count, fast bitfield subfield extract, etc.
One of the big easy advantages of NPs was direct to L2 cache I/O (instead of PCI to northbridge to DRAM to cache miss to cache). Intel now has this. NPs do still have hardware schedulers and other features, but Intel has much stronger CPUs.
Anyway, I hope Cavium succeeds with their Thunder. Perhaps there is a large enough "anything but Intel" market for them in the microserver world.
Cisco, Juniper, et al have built/are building a lot of networking gear around Cavium and Broadcom multicore MIPS parts. Intel is muscling back in to the arena, though.
The IXP was an interesting family, but I recall the toolchain being less than pleasant - although at that time the MIPS vendors weren't covering themselves in glory either.
Cavium was not yet sharing eval parts when I left this field, just a bit before Intel cancelled their NXP line (which was fun to play with).