Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
washadjeffmad
on Aug 7, 2023
|
parent
|
context
|
favorite
| on:
What's new in Llama 2 and how to run it locally
-t 32
Use a maximum of the number of physical cores and then scale down.
cfn
on Aug 7, 2023
[–]
Thanks, -t 32 (instead of -t 13 which is what comes as default) makes a big difference in CPU usage across all cores. Not quite 100% but all cores are above 50% with many at 100%. It speeds up just a tiny bit the eval t/s to 3.3 (from 2.9).
Consider applying for YC's Winter 2027 batch!
Applications
are open till November 2.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Use a maximum of the number of physical cores and then scale down.