I just finished overhauling our speculative decoding implementation for Mixlayer, so maybe I can help.
I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediate tokens even during prefill.
So for example, let's say you prefill the prompt "The quick brown fox" (and for the sake of simplicity, let's say each word is a single token). The model outputs a tensor that is [4, $vocabulary_size]. The first dimension is a token index into the input and the 2nd dimension assigns a probability to each token in the vocabulary. So even during prefill, we can look at the prediction logits for all of the intermediate tokens. That is, we can look at what the model would have predicted after "quick" and "brown", not just the tail token "fox".
In the single token autoregressive case, we just look at the next token prediction for "fox". But in the speculative decoding case we can use this information to compare the distribution of the draft model against the target model. In the greedy decoding case (ie, no sampling) we just make sure the highest probability token matches in draft and target. If we have sampling params like temperature and top-P, we have to apply something called Leviathan rejection sampling to the distribution. This basically allows us make sure the distribution is the same even if the exact probabilities are not and accept or reject draft tokens on that.
Something a lot of model providers don't talk about: any time an engine uses speculative decoding the throughput will depend on how much your output token distribution matches what the draft model was trained on.
The DFlash2 draft model we're using was trained on a lot of code, so if you use it in a coding agent you'll probably notice it run a lot faster (we've seen it break 300 tok/s).
FYI, I might be missing something but I think your billing system might not be working well - I'm not seeing any indication in the UI that my usage is being deducted from the $5 of free credits.
Hey Daniel! It's a bit hidden, but at the bottom of the billing page there's a "Credits" section which should show usage of any active credits and the balance remaining. The usage/billing metrics are batched/handled async so it might take a minute or so for usage to be reflected. Let us know if it feels off.
I had an R1T Launch Edition for a little over a year and it was hands down one of the best cars I’ve ever owned. The only issue I had was with the fob proximity sensor. The only reason I sold it was it was a bit too big for me.
If he had one shortly after they launched, the suspension was truly terrible. R1T was clearly where all the work went and it's like no one even drove an R1S before they released it. Thankfully due to SW updates the suspension is absolutely night and day better, it went from making me violently sick to pretty nice. Assuming dudes car wasn't just a lemon, if he had and got rid of the R1S before the suspension got better I can 100% believe he hates the car.
Only if you do greedy sampling. With probabilisitic sampling (categorical sampling), you will end up with different trajectory just “mathematically equivalent”.
Not OP, but you can influence how deterministic your LLM behaves using the temperature setting. The neural network doesn't directly output tokens, but logits which are then converted to probabilities and then a token is chosen at random, unless the temperature is 0 (i.e. greedy, we just always pick the most probable token without any randomness). All speculative decoding methods have to "commit" to a token though even when they don't know the actual logits of the full size NN yet. The question then is (and I don't know the answer): how do the common inference engines behave when the speculation landed on the most probable token, but the random choice still doesn't land on it? You can imagine that in the interest of performance as long as we stay reasonably inside the probability we just go ahead with the speculation. Not sure if thats implemented like that though.
Edit: I just looked up the math, and actually the idea of speculative decoding is done in a clever way that fully preserves the probability distribution while still maximizing the acceptance rate of draft tokens. So I would have to disagree with OP and say that no, non-greedy sampling doesn't influence the trajectories.
Yes, it doesn’t impact the probability distribution due to verifier. However, remember how you use PRNG and effectively due to the drafter is sampled from a different distribution initially, a separate rejection sampling won’t be able to recover what the “old PRNG” would choose in a “without drafter” case. Hence in my original post, it is about different trajectories you will end up with, not the correctness of each stochastic sampling.
If you assume that the RNG generates true randomness, then the two are identical. Only if you care about the determinism of the RNG (for example you want to use identical seeds and get the exact same generation between the two) it makes a real difference.
Correct. I am trying to explain why even it is "exact", the generated text is different from the with / without DFlash2 runs, and potentially why the DFlash2 run will contain the invalid Python syntax.
If the underlying probability distributions are the same, then DFlash can lead to an invalid Python Syntax iif the autoregressive process could have generated one if the random sampling picked a different token.
If a model can output a “wrong” sequence with a certain probability p, then Dflash can also output the wrong sequence with the same probability. They wouldn't necessarily produce the same output from the same seed, but speculative decoding shouldn't be able to produce anything that the autoregressive model couldn't also produce when using a different seed.
It will contain the same or different syntax with or without it. Also multiple runs without DFlash2 will contain the same or different syntax. And multiple runs with DFlash2 will have the same or different syntax with the same probability. DFlash2 literally has no influence (unless buggy). The difference is purely caused by the randomness.
We're getting to the limit of my understanding, but I believe most Blackwell users still usually run FP8 passes through the transformer engine - they'll just store weights at NVFP4. Nvidia has model-specific stabilization recipes for NVFP4 end to end, but they're taking fixes all the time.
Nvidia says Rubin should have fewer stability problems training with FP4 because of hardware changes - "adaptive compression". There will still be outlier instability inherently, but something they're designing in reduces the cost of managing it.
But yeah, grain of salt - we haven't seen this in practice.
I'm also puzzled by that statement. The issue with training is (as I understand it) one of precision and the associated numerical stability. You need enough bits in order for backprop to function correctly.
Of course there are techniques such as quantization aware training but I don't understand why a datatype would work for inference but not for that.
You can also abandon backprop entirely but that comes with a whole host of tradeoffs and again why would it work for inference but not for whatever alternative training regime you selected?
If you're in SF and weighing this decision, it's easy to get tilted in the buy direction because the rental stock is so horrific. Landlords have very little incentive to update properties or provide basic amenities that people take for granted in other major cities (good luck getting a washer/dryer).
With the 3.5 release, the Plus model was just a rebrand of the open weight 397B. But I suspect that will change going forward. They haven’t released the weights for 3.6 but they did make it available through a few US providers.
I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediate tokens even during prefill.
So for example, let's say you prefill the prompt "The quick brown fox" (and for the sake of simplicity, let's say each word is a single token). The model outputs a tensor that is [4, $vocabulary_size]. The first dimension is a token index into the input and the 2nd dimension assigns a probability to each token in the vocabulary. So even during prefill, we can look at the prediction logits for all of the intermediate tokens. That is, we can look at what the model would have predicted after "quick" and "brown", not just the tail token "fox".
In the single token autoregressive case, we just look at the next token prediction for "fox". But in the speculative decoding case we can use this information to compare the distribution of the draft model against the target model. In the greedy decoding case (ie, no sampling) we just make sure the highest probability token matches in draft and target. If we have sampling params like temperature and top-P, we have to apply something called Leviathan rejection sampling to the distribution. This basically allows us make sure the distribution is the same even if the exact probabilities are not and accept or reject draft tokens on that.
reply