What is an example of an encoder-based transformer?

• A. Llama from Meta.
• B. GPT-4 from OpenAI.
• C. T5 from Google.
• D. BERT from Google.

Answer: (D)

An encoder-based transformer uses only the Transformer encoder to build contextual representations of the input, focusing on understanding and representation rather than generating new text. BERT exemplifies this approach: it processes the whole input with a bidirectional encoder, learned through masked language modeling, and produces rich representations that downstream tasks (like classification or QA) use directly. The other models operate differently: Llama is a decoder-only model designed for generation, predicting the next token in sequence; GPT-4 is also decoder-only and optimized for autoregressive text generation; T5 uses both an encoder and a decoder, making it an encoder–decoder model rather than encoder-only.