Parameter-efficient adaptation
Adapt a big frozen model by training only a small number of new parameters.
| Concept | Introduced by | Years | Papers |
|---|---|---|---|
| Adapter modules Insert small trainable layers into a frozen network, one set per task. | Adapters | 2019–2021 | 2 |
| Prefix tuning Optimise a few continuous 'virtual tokens' prepended at every layer of a frozen LM. | Prefix-Tuning | 2021 | 2 |
| Soft prompt tuning Learn only input-level prompt embeddings; it rivals full fine-tuning at scale. | Prompt tuning | 2021 | 2 |