What role does attention play in transformer models?

• A. It helps models focus on relevant parts of input sequences for better contextual understanding.
• B. It ensures the model processes all tokens equally.
• C. It sorts tokens alphabetically before generation.
• D. It collapses all attention scores to a single value.

Answer: (A)

Attention in transformer models lets the model decide which parts of the input to emphasize when forming representations for each position. By computing weights over all tokens, it creates a context vector that blends information from the most relevant tokens, enabling better understanding of meaning and relationships, including long-range dependencies. In self-attention, each token uses a query to compare with keys from all tokens and then mixes the corresponding values according to those learned weights. Multiple attention heads let the model capture different kinds of relations at once, enriching the representation. This approach focuses on what matters rather than treating every token the same, and it does not sort tokens or collapse all scores into a single value.