Perhaps instead you are referring to the program graphs and whether they are better represented as hyper graphs [0] with certain regular edges within a 2D layer and hyper edges between 2D layers?
If so, good question and I am not sure if the answer, I agree that it would probably be a helpful abstraction layer to assist with certain data locality and reduce data movement + congestion issues.
[0] https://lasttheory.com/article/what-is-a-hypergraph-in-wolfr...
Curios what you mean by 3D and also how 3D is used. Assuming something like [0] I can pretty confidently tell you that it is not 3D as it is a low power microcontroller and this technology is mostly used in large expensive HPC/AI chips also afaik 3D stacking of logic die is not really a thing (if anyone knows counterexamples pls provide) it is much more common for stacking memory die ex HBM. As for your proposed coprocessor that actually might benefit from 3D integration with the trad cpu as >> # of interconnect / memory channels could allow you to route data more directly to processing elements. Something like this is proposed with “2.5D” stacking in [1] where HBM (3D) is connected to an FPGA with 128 channels.
See figure 1 in [0] operation execution is just 0.1pj of 70pj required for a 32 bit int addition… how do you think I was applying amdahls law?
Edit: I was using amdahls law as it applies to parallel processors. However in terms of energy usage …
While it is true that cheap and expensive FPGAs exist, an FPGA system to replace TPU would not use a $0.50 or even $100 FPGA it would use a Versal or Ultrascale+ FPGA that costs thousands, compared to the (rough guess) $100/die you might spend for largest chip on most advanced process. Furthermore, overhead of FPGA means every single one my support a few million logic gates (maybe 2-5x if you use hardened blocks), compare to billions of transistors on largest chips in most advanced node —> cost per chip to buy is much much higher.
To the second point, afaik, leading edge Versal FPGAs are in 7nm, not ancient also not cutting edge used for asic(n3).
Good questions, below I attempt to respond to each point then wrap it up. TLDR: even if TPU is good (and it is good for Google) it wouldn’t be “almost as good a business as every other part of their company” because the value add isn’t FROM Google in the form of a good chip design(TPU). Instead the value add is TO Google in form of specific compute (ergo) that is cheap and fast FROM relatively simple ASICs(TPU chip) stitched together into massively complex systems (TPU super pods).
If interesting in further details:
1) TPUs are a serious competitor to Nvidia chips for Google’s needs, per the article they are not nearly as flexible as a GPU (dependence on precompiled workloads, high usage of PEs in systolic array). Thus for broad ML market usage, they may not be competitive with Nvidia gpu/rack/clusters.
2)chip makers with the best chips are not valued at 1-3.5T, per other comments to OC only Nvidia and Broadcomm are worth this much. These are not just “chip makers”, they are (the best) “system makers” driving designs for chips and interconnect required to go from a diced piece of silicon to a data center consuming MWs.
This part is much harder, this is why Google (who design TPU) still has to work with Broadcomm to integrate their solution.
Indeed every hyperscalar is designing chips and software for their needs, but every hyperscalar works with companies like Broadcomm or Marvell to actually create a complete competitive system.
Side note, Marvell has deals with Amazon, Microsoft and Meta to mostly design these systems they are worth “only” 66B.
So, you can’t just design chips to be valuable, you have to design systems. The complete systems have to be the best, wanted by everyone (Nvidia, Broadcomm) in order to be in Ts, otherwise you’re in Bs(Marvell).
4. I see two problems with selling TPU, customers and margins. If you want to sell someone a product, it needs to match their use, currently the use only matches Google’s needs so who are the customers? Maybe you want to capture hyperscalars / big AI labs, their use case is likely similar to google. If so, margins would have to be thin, otherwise they just work directly with Broadcomm/Marvell(and they all do). If Google wants everyone using cuda /Nvidia as a customer then you massively change the purpose of TPU and even Google.
To wrap up, even if TPU is good (and it is good for Google) it wouldn’t be “almost as good a business as every other part of their company” because the value add isn’t FROM Google in the form of a good chip design(TPU). Instead the value add is TO Google in form of specific compute (ergo) that is cheap and fast FROM relatively simple ASICs(TPU chip) stitched together into massively complex systems (TPU super pods).
Sorry that got a bit long winded, hope it’s helpful!
This also all assumes that there is excess foundry capacity in the world for Google to expand into, which is not obvious. One would need exceptionally good operations to compete here and that has never been Google's forte.
