I ran into the same problem as you, so I ended up by coding a local app that is very similar to Wispr Flow, but uses the small english Whisper model on my low-end Windows laptop.

It is still a quite fast. In fact, I just typed this in using this app.