LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu, Yelong Shen, Phillip Wallis et al.
ICLR 2022 · arXiv:2106.09685 · paper page

LoRA adapts pretrained language models through trainable low-rank weight updates while retaining the original model parameters.

Problem

Full fine-tuning requires gradients and optimizer state for the pretrained weights, followed by storage of a separate adapted model for each task. LoRA examines whether task-specific changes can be represented in a smaller parameter space. The original work concerns language-model adaptation rather than a GUI policy or visual grounding architecture.

Contributions

A low-rank parameterization constrains the update to a pretrained weight matrix while leaving the original matrix frozen during adaptation.

The update can be merged into the pretrained weights for inference, and task-specific adapters can be stored separately from the shared base model.

Empirical studies examine where to apply the updates, their rank, and the relationship between the learned update subspace and the pretrained weights.

Method

For a selected linear layer, LoRA represents the weight change as the product of two smaller matrices. The layer adds the low-rank branch's output to the frozen base transformation. One matrix starts at zero, so the adapted model initially reproduces the base layer; a scaling factor controls the update magnitude.

Backpropagation updates the low-rank matrices while the base parameters remain fixed. The paper focuses on attention projections in Transformer language models. Rank and the selection of adapted matrices determine the trainable parameter budget; the method does not require reducing the size of the underlying pretrained model.

At deployment, the matrix product can be added to the base weights, preserving the original layer structure. Alternatively, adapters can remain separate to support switching between tasks. The released loralib implementation demonstrates replacing selected layers, marking adapter parameters as trainable, and saving adapter state without duplicating the complete pretrained checkpoint.

A frozen weight matrix maps input to output while a parallel low-rank pair of matrices contributes a trainable update. — Figure 1 from the paper: LoRA adds trainable low-rank matrices alongside frozen pretrained weights. (Figure rights retained by the original authors; source)

Evaluation

Language-understanding experiments use GLUE with RoBERTa and DeBERTa. Generation experiments cover E2E, DART, WebNLG, WikiSQL, and SAMSum with GPT-family models. Comparisons include full fine-tuning and other parameter-efficient adaptation methods.

The evaluation considers task-specific prediction or generation metrics alongside trainable parameters and training resource requirements. Ablations vary attention matrices and rank, while subspace analysis investigates the structure of learned updates. No GUI-navigation benchmark is part of the original evidence.

Open this note in the interactive notebook (comments, hooks) → · All notes