How does LoRA work in practice?

• A. It modifies all layers with large gradient updates.
• B. It freezes the base model and adds small adapter weights to transformer layers.
• C. It replaces attention mechanisms with new modules.
• D. It doubles the number of parameters to improve capacity.

Answer: (B)

The main idea behind LoRA is to adapt a large pre-trained model without retraining all of its parameters. It does this by freezing the backbone and adding small, trainable adapters inside the transformer layers. Practically, the update to a weight matrix is expressed as a low-rank addition: deltaW = A B, where A and B are much smaller than the original weight matrix and are the only parameters learned. The original W stays fixed, so training focuses on these tiny adapter matrices, dramatically reducing the number of trainable parameters.

These adapters are usually inserted into key parts of the transformer, such as the attention projections (queries, keys, values, and outputs) and sometimes the feed-forward blocks. During training you optimize A and B, and at inference time the effective weight is W plus deltaW, so the model behaves as if it had learned the updated weights but with far less computational overhead and memory use.

Why this approach fits practice well is that it preserves the base model’s pre-trained knowledge while enabling task-specific adjustments. It avoids large gradient updates across the whole network, does not replace core mechanisms with new modules, and does not simply double the parameter count; instead, it adds compact adapters that shift model behavior in a targeted, efficient way.