Which describes how LoRA modifies transformer layers during training?

• A. It removes attention mechanisms from transformer layers.
• B. It updates all weights in the transformer layers.
• C. It adds small adapter weights while freezing most of the base model.
• D. It shrinks the model by removing layers.

Answer: (C)

LoRA trains transformer layers by inserting tiny, trainable adapter components into the layer while keeping the original pre-trained weights fixed. The idea is to learn a small set of additional parameters that adjust how the layer behaves for a new task, rather than updating the entire large model.

During training, only these adapter parameters are updated; the base model stays unchanged, so the knowledge learned during pre-training remains intact. Because these adapters are low-rank matrices, they add far fewer parameters than retraining the whole model, making fine-tuning more parameter-efficient in terms of both memory and computation.

This approach isn’t about removing parts of the model or shrinking its size. It’s about augmenting the existing layers with small, trainable modules that capture task-specific adaptations while preserving the backbone’s original capabilities. You can then reuse the same backbone across multiple tasks by swapping in different adapters.