Awesome, thank you! Even though it's not what I was hoping for. If you're still set up, you could try with Chrome on both ends, since a single Firefox participant is known to degrade the quality for the whole conference. (I'll try to find a reference once I'm at my desk.) I suppose if the rpi is at 100% cpu that's unrelated, though.
Ok, so I gave it a test with Chrome on the Laptop and Chromium on the Pi. It starts at 720p ~10fps and then drops to 270p ~30fps after a few seconds (I'm not actively cooling the Pi). This is a slight improvement since it didn't hit "HD" at all in the first test. Note the camera connected to the Pi is only 720p so it's at its top resolution.
I can add active cooling to the Pi and test further once I find the fan I have lying around here somewhere.
EDIT: Added active cooling and increased video memory from 76MB to 512MB (although I think it's capped at ~400MB if I remember correctly), I'm now getting encode of 640x360 @ 15FPS on the Pi, so a slight improvement.
Edit: The issue I mentioned appears to be fixed for a while: https://github.com/jitsi/jitsi-meet/issues/5439 More discussion here: https://github.com/jitsi/jitsi-meet/issues/4758