Its a pattern that works well in VFX, It has the advantage over something like isilon in that hotspots are isolated to individual servers, not across the namespace. So if one of your git stores is being hammered, you can migrate hot/cold repos to other servers fairly simply. Also if one of the server dies, it has a limited blast radius.
The problem with things like ceph-fs (and lustre and to a lesser extent, GPFS, although its not entirely comparable) is that the metadata store is your weak link. Ceph scales great if you have loads of large files where you're read/writing in parallel. (ie pulling thousands of PAR files or images, videos or binaries) it scale almost linearly with the number of object stores. It also works well when your writing to the middle of a file. (far fucking better than s3 like systems)
git is monster metadata eater. Everything git wise is a metadata lookup. That means that when you are running thousands of concurrent git ops on a distributed filesystem, your object throughput will fall off the floor. so you could have 100 ODSs all on 100 gig network with massive nvme stripes, but your global throughput will be shite because your MDS is the limiting factor. You can add more metadata servers, but then ceph is choosing how to shard, not you.
either way, deleting a large git repo, then all your metadata operations start crawling.
This means that you need to think about doing optimisations like keeping git inside a tar or some other container container that are pulled out, loaded in ram, operated on, and forced back as a binary blob. the result means that your thousands of metadata ops are reduced to two or three, and your back to being network bound.
The problem with things like ceph-fs (and lustre and to a lesser extent, GPFS, although its not entirely comparable) is that the metadata store is your weak link. Ceph scales great if you have loads of large files where you're read/writing in parallel. (ie pulling thousands of PAR files or images, videos or binaries) it scale almost linearly with the number of object stores. It also works well when your writing to the middle of a file. (far fucking better than s3 like systems)
git is monster metadata eater. Everything git wise is a metadata lookup. That means that when you are running thousands of concurrent git ops on a distributed filesystem, your object throughput will fall off the floor. so you could have 100 ODSs all on 100 gig network with massive nvme stripes, but your global throughput will be shite because your MDS is the limiting factor. You can add more metadata servers, but then ceph is choosing how to shard, not you.
either way, deleting a large git repo, then all your metadata operations start crawling.
This means that you need to think about doing optimisations like keeping git inside a tar or some other container container that are pulled out, loaded in ram, operated on, and forced back as a binary blob. the result means that your thousands of metadata ops are reduced to two or three, and your back to being network bound.
So yeah, no I stand by my original position.