I often switch between feature branches, when working on more than one project in a repo with multiple related modules. If there's a change near the top of that dependency graph, I'm forced to not exactly rebuild from scratch, but still to rebuild quite a lot.
Right now I'm at the very same situation. Yes, it is frustrating. This is why things close to the top of the dependency graph should be small, well-tested, and rarely need changes. But when you still need to troubleshoot them, there's no way around recompiling a lot of stuff if you want these static guarantees :(
In my case it's not even a language proper; I was using JavaScript which gives you near-zero static guarantees.
I was fixing an issue in one common library; properly testing changes required rebuilding and restarting a number of containers. Unit tests only tell you so much; you need proper integration tests to see how certain things interact.
If I were used a statically typechecked language (e.g. TypeScript), I could have eliminated 50%, or maybe 75% of the testing, because the compiler would check things for me before runtime. It would be drastically faster to localize and fix the bug even if the compilation increased build times 10x.