It uses a custom delta compression algorithm, based on libXDiff. You can see that output using "git diff --binary". I believe these are the relevant parts of the source code:

https://git.kernel.org/?p=git/git.git;a=blob;f=diff-delta.c;...

https://git.kernel.org/?p=git/git.git;a=blob;f=diffcore-delt...