Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM.
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.
> --spec-draft-n-max 5
5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is.
I think when tuning 3.6, I saw more like 33%? But not 100% sure.
I wonder if we could take a page out of the Solar power book and do MPPT but for draft count.
Constantly sweep through possible values to find the best result for the current conditions.
Though it might turn out that that doesn't offer any benefits. Has it been tried yet?
___
EDIT:
LLM pointed me towards this thing I'm not going to read
Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?
This is all a pretty active area of research, both adaptive drafting, different draft approaches (DFlash etc), I expect to see really big improvements land in the popular runtimes this year.
Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33
The unweighted average is 4.1
So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5.
Also, cool that it carries over from qwen3.6 like that.
I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.
How many tokens/second you getting? I have the same CPU but a 5070 and only 64 GB of ram. I just got llama.cpp built and am now hitting a whopping 5-6 tokens/s.
I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.
Problem with that is I think that it quickly devolves into cargo culting, nonsense and noise.
Arguably, what I am doing is also very very close to that, with the only difference being that I am somewhat less of an idiot than the average internet dweller you'd get on such a site. Or rather a different flavor of idiot.
Ideally, the people building the tools build them in a way that just does the right thing - which I am confident that llama.cpp does or will do in the future.
So you encode that knowledge not in language and online comments but in code and with a filter for actual expertise.
And, frankly, there's really not all that much to it. It's like maybe 3 parameters to play around with.
The valid solution space is pretty small, but people will want to make it "theirs" regardless, so you get non-solutions just so that everyone could also be a part of it. The usual social dynamics foo.
I've found that once you factor in multiple GPUs things can get complex quite quickly because the default packing routine in the LLM runners tends to be very coarse resulting in substantial amounts of VRAM wasted. More so if you start running drafters and multiple models at the same time.
Let me return the favor:
Deepseek V4 Flash 0731 Q8
This is on a 4x3090 box with 256G RAM and a 5955WX TR; the CUDA drivers are patched to allow the 3090's to talk to each other. MB is a WRX80E-SAGE (which has 7 slots at x16), BAR > 4G on and PCIe BAR resize set to AUTO in the bios. OS is Ubuntu 25.04.
I'm still working on an optimal packing for the draft variation on this theme but that gives all kinds of issues. The 1,000,000 context is very nice to have.
still could! there's the network site https://ai.stackexchange.com/ just need people to ask questions and to downrank accepted answers older than a year (that might not be possible though).
I can get 128k context on a 5070ti with 16 GB of VRAM (using the Unsloth 2-bit quant[0]). This is via a .bat file on Windows 11. I'm getting about 50-60 tokens/second and the quality is much higher than Qwen 3.6 27B. I'm using llama.cpp[1]:
1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder.
2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`.
3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window.
You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored).
Congratulations, you're now running Qwen 3.8 27B.
Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB
Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!
According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage.
But maybe someone else has experience to share there
just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort.
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
I've worked with a good few types of managers of the years, none would take ownership of my work. The end result? Sure, for shareholders/this managers that is fine.
But that's not the same as me being needed to be on a call with integration teams and having to run the call. It's just a layer of abstraction for management. I own the creation side of the product.
this topic always goes into this level of ridiculousness. Your question can't even be answered. We should focus more on encouraging discussion, not asking unanswerable questions. It wont bring anything other than points of both sides with no objective correct answer.
I generally agree and expect this to be the case from a legal perspective.
Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.
So if I hire an artist and am a motivated individual, have an idea for a painting, have background knowledge about paintings and have taste in paintings and can offer a critique of the painting as the artist paints it, then somehow I created the painting?
Nothing absurd about that. What do you think an "Executive producer" is? A "Director" ? Does Peter Jackson get credit for creating the Lord of the Rings Trilogy films? Christopher Nolan for his films? But did he make them ? No, it was the collective effort of thousands of individuals all working under their direction.
Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.
What does it matter who receives more credit? The producer produces the film and the director directs it. If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it.
If there were multiple people involved in the creation of a film I helped to create, I cannot factually say I created it. Just like if someone builds something using code generated by AI, they can't factually say they created it.
There's a bit more nuance to it though, I think, because traditionally there has been more of a firm line between humans and tools.
For example, most people would agree with this line you wrote: "If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it." A pedant could say "woah hang on, that's ignoring the fact that actually the iPhone is the one recording the images that make up the film, how can the human get all the credit", yet nobody would actually make that argument when discussing who created the film.
Generative AI is pretty much the first time (maybe there are some niche contradictions to this claim?) we consider a tool to be contributing enough creativity to the process that we don't all agree "only one person was operating this tool, so that person is the sole creator" - but some people DO still hold that line, and do consider the human who wrote the prompt to be the creator.
And I don't think there's any objective technical metric we can use to say who's right, it comes down to our collective judgement deciding where the line is.
That’s an interesting way to look at it, and still generous to the llm.
By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.
Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.
Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”
I think in that case it's fair to say you created the painting with the artist, even if the artist should get majority credit. I don't like the analogy though, to me it feels more like you're a project manager directing a team of genius but single minded interns.
I've never encountered a project manager who has said: I built this thing, after a project has wrapped up. I have encountered ones who've said I managed this project.
I've heard some variant of "we built this thing", which I think is valid. I'm not sure how I feel about anthropomorphizing a coding agent at this point though, which is what you're implying.
I just read through a couple of your posts that weren't dead or buried, and it seems like you're pretty anti-AI. You should really start to have an open mind towards it. It's going to be the future (if it isn't already), and as you continue to get older, you're going to really wish you spent your time right now learning and embracing the technology instead of being so against it. A lot of the skills and things that you're holding on to right now might not be relevant by then, but you'll be at a disadvantage from not keeping up with the industry and need to play catch-up.
> I'm not sure that typing messages to a chat bot requires much catching up
Lol, yeah I'd wager AI tooling skills are less relevant than knowing your way around a shell. It doesn't help that all of the AI tooling has event more churn than js libraries and package management systems!
I think there's probably some value in understanding how LLMs work, but beyond that it's the same boring skills that matter the most... Critical thinking, design chops, attention to detail, perseverance, etc.
He still didn't create the paintings, he offered instructions / feedback to the artists that created them. Whether or not he is credited with authoring them is an entirely separate issue.
There have been plenty of workshops where artists hire assistant painters while maintaining authorship over the works themselves, from Rembrandt to Warhol to Hirst.
Sure - and a factual statement would be, these artists hired assistant artists to help them create their paintings. Just like it would be factual to say that the person directed a LLM to build an app for them.
The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.
If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.
Does anyone know how to get this working with Claude Code via llama-server? I'm getting a jinja template error about the system prompt not being the first message.
You could run the 2-bit quantization at about 10GB for the weights. You'd probably also need to quantize context to get a decent-sized context. The Unsloth model card shows the 1-bit quantization doing seemingly shockingly well on a task (but there is notable loss in model quality for each bit you give up below about 6 bits). An MoE model can partially reside in system RAM without catastrophic performance loss, but a dense model like this one cannot. You have to run it entirely out of VRAM, or it'll slow to a crawl.
Yes. It won't be as fast as fitting the whole model into VRAM but llama.cpp defaults are pretty smart about GPU/CPU splits these days. Just YOLO it with `llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL` and it'll definitely at least run.
You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.
For the things it does, what tool is better than ffmpeg? Really struggling to see the cargo cult angle. Similar for llama.cpp, as it is literally the only framework I can get to run on my multi-Radeon rig. It is the most portable runtime out there.
It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
As opposed to loading it up in RAM + VRAM? Pretty much always better to split it up to multiple GPUs. My priorities are load up all available VRAM, then offload MoE experts to RAM (if possible), then offload other layers.
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Yes. I can split a model like this across 3 GPUs (a 1080 with 8GB and two Titan Vs with 12GB), and it's much faster than running it on 36 CPU cores. As long as it fits in aggregate VRAM, it seems very advantageous to do so.
That's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... .
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
Depends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
I haven't tried this either but I'm guessing if you could pool the GPU memory over whatever the kids are using these days, I think it was SLI back in my day. The GPU memory should still be faster than the RAM?
Unfortunately I checked, SLI doesnt work for this situation. Because the program loading the LLM uses CUDA library, which doesnt account for/takes advantage of SLI for this purpose at least.
Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command.
I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.
What makes you say that it would be hard to do that?
It's long, I guess, but not cryptic.
You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.
You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers.
I think at least. I've never tried. My hubris stopped me from trying auto configs.
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Why would they wish to handcraft this ? That is what agents are for ?
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future.
Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore.
It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines.
Lower level API/interfaces should not be a barrier for people if they are apply framing that way. More and more people are thinking agent native so this is not really a issue.
The alternative is to label it AI. That is distasteful for some of us, Intelligence is much stronger than automation in our minds so we avoid the term and try to describe it differently .
This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX.
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.
Sure, but shouldn’t the programs to run the LLMs go “the user has this much vram and the model is this size, so I’ll start with sensible defaults based on that”?
I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived.
There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me.
Even this command above, it's not clear where op got the model from. So I'm with yah.
For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..
Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention and it will handle downloading locally for you.
But you're assuming I'm using the Olama studio. This model as far as I see doesn't have a gguf download.. Unless I'm missing something on the page.
If I want to download the model myself, it's not clear. I thought it was supposed to behave like a package manager. But even in nuGet I can download a zip of the package.
What on earth are you talking about? llama.cpp != Ollama. You can (and should) just use llama.cpp directly. Upstream llama.cpp can take the shorthand huggingface path and automagically download it into a cache folder as part of the launch. Have you read any of the docs?
Excuse me, but thats a direct link you've just sent. I asked where I can find the links. I like to believe in the source of truth.
They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Do you see how steep the barrier for entry is to do anything right ?
unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that, and I want AI to be a better tool.
> Excuse me, but thats a direct link you've just sent. I asked where I can find the links. I like to believe in the source of truth.
You asked where to find the GGUF files of this model for direct download and I provided it. Almost all useful model files that can be downloaded are hosted on Huggingface.
> They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
I went to Huggingface, went to the Unsloth org, as they tend to be the best, went to the Model page, and went to the "Files and versions" tab.
> Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Unsloth AI is a very popular, highly reputable organization that takes upstream model files, performs some optimization, and provides models in various formats. Apart from speed tweaks, they do not modify the models. They also provide useful benchmarks, copious documentation for local execution, and a Studio application for easy execution and post-training of models.
> Do you see how steep the barrier for entry is to do anything right ?
No. Searching for this information is not difficult. The llama.cpp documentation and guides that Unsloth provide are all you need. Search engines can take you further if you want.
> unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that
Qwen also provides models in GGUF format on Huggingface, but they will not be as performant. Even when first-party GGUFs are available, most people will prefer quants from Unsloth or a few other popular optimizer accounts.
> I want AI to be a better tool.
Best of luck. Your attitude and unwillingness to even try and learn on your own when people have tried helping have burned my good will, and this is as far as I'm willing to carry you.
I gave you a direct link to where you can get a download, and explained why I chose to link you where I did. I can't post screenshots here drawing you a map of how to use a website. I don't work for Huggingface or any AI company and can affect no changes to how intuitive any of it is, and think is easy enough already.
Huggingface, Unsloth, and llama.cpp all have documentation you can follow that will exceed anything I can tell you here. LMstudio, Lemonade, or Ollama might be even easier for you to use. Take my suggestions or don't.
Start by copying the command line from the Unsloth guides.
You don’t need to fine tune all of those parameters to get started.
It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.
> Ollama is confusing and doesn't seem to support Qwen3?
Typing “Ollama qwen3” into Google takes you right to this page:
If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
on Qwen's official docs: https://qwen.readthedocs.io/en/latest/run_locally/ollama.htm.... It's not like I just made it up. Of course I searched "ollama qwen3" and saw what you linked, but that doesn't mean it "works". I have other things to do besides to try a bunch of poorly documented and executed tools just to see if it works or not.
I guess the TLDR is that I'm stupid or lazy. Also, everyone is responding about how easy it is, and yet, it's apparently so easy that it's hard to document well.
There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.
I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.
While you probably have the audacity to check the commands and code that your favorite LLM generates to complete this task, i'm pretty much certain that at some point in time, due to this pervasive mindset of "just let an AI agent do it," we'll have a front page news story of someone who had their family photos on their NAS uploaded to some public space, their security cameras connected to a "live porn for lolz" webring, and perhaps even their crypto wallets and bank accounts pilfered, all because their favorite LLM got exploited by some prompt injection attack while trying to be helpful doing system administration tasks.
I remember there was a short story in BYTE Magazine about a similar kind of scenario way back when, I think at least 30 years ago, long before LLMs and AI agents became a reality.
Hmm I'm still struggling with the downloading of all the models I want to try locally, so I might be out of the loop, but all of this, including tuning (to your local HW), feels like it could be automated. I'm old though.
> Every big company in the world uses a network proxy.
It's becoming more rare, now.
A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.
The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
I would love to see more data on that because I've seen it constantly. There is more isolation maybe where you can do whatever on 'open' network, but always some kind of proxy/vpn connection for hitting anything sensitive.
The operlap is there.. But I would be worried if it was just flat out taken away from secure managed connections just because of AI.. Again, would love to see the numbers of your assumptions.
Woa TIL. I thought that was somehow long solved at the OS level or with VPNs or something like that (no idea exactly how, I'm sure just I'm misunderstanding something basic).
Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz