for layer parallelism (e.g. to get more vram) the bandwidth between layers is es... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

nullc 35 days ago | parent | context | favorite | on: Qwen 3.8 27B

for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!

ericd 35 days ago [–]

Good point. It's much more of an issue when running dense models with tensor parallelism. In that case, I'd look for an MoE model instead.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact