The generation is done by... prediction. It is making a guess based on weights (statistical likelihood) plus a pinch of wobbliness.

It's worth reflecting on the original domain jargon for selecting the next token: completion.

I know how LLMs work. I understand the output projection and logits and sampling just fine. I don't think we should call what is happening prediction because it is a misleading term with a lot of baggage associated with claims that LLMs are just statistically modeling language and "guessing" tokens without any internal representation of meaning. Since your reply is very short I can't tell whether you mean it in that sense or not.

And yes, I understand "prediction" is a standard term in the field. I just think it is a bad one and we should stop using it.