Not if your use case needs speed. For one of my products I can't use an LLM that... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

voiceeh 18 days ago | parent | context | favorite | on: Mercury 2.5 LLM hits 770 tokens per second

Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.

hansvm 18 days ago [–]

If it could output 1k tokens per second but needed 4 seconds to produce the first batch of 4k, would that not be viable?

Consider applying for YC's Winter 2027 batch! Applications are open till November 2.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact