Which statement best describes autoregressive generation in language models?

• A. Global optimization over all tokens
• B. Generating images
• C. Classifying inputs
• D. Predicting the next token from previously seen tokens

Answer: (D)

Autoregressive generation in language models is about predicting the next token given the tokens seen so far. The model learns a conditional distribution for each position: P(next token | all previously generated tokens). Generation proceeds step by step, choosing a token and feeding it back as context for predicting the following one, often by sampling or selecting the most probable option. This sequential factorization is what makes the process autoregressive: each step depends on the history of past tokens, not on future ones.

This differs from the other ideas: global optimization over all tokens would imply optimizing the entire sequence at once rather than predicting step by step; generating images isn’t specific to language modeling; and classifying inputs is about assigning a label rather than producing a coherent token sequence.