Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.

If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.



For those us us who don't know, what do those parameters do and why are they better?


Temperature, top-up, top-k, min-p all control which token the model predicts next and how likely it is to select one token over the other.

You might understand this as "The capital of France is..." and the model isn't always going to select "Paris". Sometimes it will start a descriptive sentence or even get the answer wrong.

That selection of the next token is what these settings control, and lots of sub-optimal selections compound over time to produce a junk response.


I broadly knew that about temperature, but lack the background in machine learning/statistics to differentiate top-n-sigma from top-k/top-p.


Top-K: example setting 20. Select only from the 20 most likely tokens.

Top-P: example setting 0.9. Select tokens whose probably accumulates to this number. So say you have tokens with 0.7 then 0.2 then 0.1, the last will not be selected because the first two tokens already accumulated to >=0.9.

Min-P: example setting 0.05. Don't select tokens less probable than this value. So a token with 0.1 would be considered, a token with 0.01 would not.

The purpose of all of these is to exclude very unlikely next tokens.


Min-p is specifically "Don't select tokens less probable than a multiple of the top token's probability" with min_p of 0.1 multiplied by an example top probability of 0.3 being 0.03 as the truncation at that time step.

Source: One of the min_p authors


Thanks, today I learned!



You might find this article relevant: https://news.ycombinator.com/item?id=49151933


I've run the inference to get the answers I linked to. If someone else does the same thing, that involves extra energy. If I read your conversation instead of generating my own, then that's one less tree that has to be chopped down.

Until we go advanced geothermal or we crack fusion, energy is dirty. Read my inference or link me to yours so we don't boil the planet.


The Qwen team published the same sampler settings for 3.8 and presumably they used those while testing on benchmark. Do you believe they could have achieved higher result with top-n-sigma?


Diverging from the sampler used in RL training is not good for long multi-turn results-- it's a great way to knock models into reasoning loops that wouldn't otherwise.


Peer reviews NeurIPS caliber paper to prove that? Because I can show you one titled "Long context generation is a sampling problem"...


Show us, we're curious. Did you upload to ArXiv yet?


How is terrible settings a case of FUD?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: