What does it mean when LLMs are autoregressive?

• A. They regress to the mean to reduce variance
• B. They can only handle labeled data
• C. They generate text by predicting the next token based on previous tokens
• D. They can output text only after the entire input is known at once

Answer: (C)

Autoregressive means the model builds text one token at a time, with each new token chosen from a distribution conditioned on all tokens that came before it. During training, it learns the probability of the next token given the preceding sequence, effectively P(next token | previous tokens). When generating, you start with a prompt, pick or sample the next token from that distribution, append it, and repeat to extend the text. This left-to-right, history-conditioned process lets the model produce fluent and coherent sequences because every prediction depends on the growing context. By contrast, approaches that predict many tokens in parallel aim for speed but don’t rely on the same step-by-step conditional generation.