sort of sounds like "project tapestry" by Yann LeCunn. Build projected data silos of highly valuable information, train in a distributed manner and share the weights upwards where they're combined and fine tuned.