On Gemma 4 12B, I am getting 220 ms per move or QS. I used it to play the Snake game locally:
prompt_eval=244 ms wall=245 ms schema_cache=hit generated=0
Move limit reached after 200 moves: score=16, length=19.
So, if a 12B dense model can offer this latency on a local old PC, then definitely you can scale it up with more powerful machines and get even lower latency.
prompt_eval=244 ms wall=245 ms schema_cache=hit generated=0
Move limit reached after 200 moves: score=16, length=19.
So, if a 12B dense model can offer this latency on a local old PC, then definitely you can scale it up with more powerful machines and get even lower latency.