It doesn’t help that, if you enable IPv6 in Debian, the network becomes extremely slow for many things. The reason? If first tries to reach all domains with IPv6, times out, then use IPv4. Of course that’s because I didn’t successfully configure IPv6 with the network, but on the other hand, IPv4 only was successful at first try.
That's not how v6 works in Debian, or really anywhere. For one thing, you can't reach a domain with v6 if there's no AAAA record, so it certainly won't be trying to reach "all domains" with v6.
If your network has broken v6 then yeah, that's gonna cause problems. That's not v6's fault, that's down to having a broken network.
Linux doesn't have a big "enable IPv6" switch, so it's not clear what you mean.
The only way I can think of getting the behaviour you describe is if you configure the network to hand out v6 addresses and routes to hosts, and then just black hole the packets. What you have then is a broken network.