What are examples of Seq2Seq transformers?

• A. BERT and RoBERTa.
• B. GPT-3 and GPT-4.
• C. Llama and OPT.
• D. T5 and BART.

Answer: (D)

Seq2Seq transformers are built as an encoder–decoder pair that takes an input sequence and produces a corresponding output sequence. The encoder processes the input and creates representations, and the decoder uses those representations to generate the output step by step, making them well-suited for tasks like translation, summarization, and question answering.

T5 and BART fit this pattern. T5 treats every task as a text-to-text problem, so you feed in input text and the model outputs the desired text, all within the same framework. BART combines a bidirectional encoder with an autoregressive decoder and is trained with a denoising objective, which helps it understand the input well before generating a coherent output. Together, they exemplify the encoder–decoder approach central to Seq2Seq transformers.

The other models are not built as encoder–decoder seq2seq systems. BERT and RoBERTa are encoder-only, great for understanding but not for generation in a single end-to-end seq2seq setup. GPT-3, GPT-4, and other decoder-only models like Llama and OPT generate text in a left-to-right, autoregressive fashion and aren’t designed as paired encoder–decoder systems.