training technique
Pre-train, then fine-tune
Train one big model on unlabelled data, then fine-tune it briefly for each task.
Drafted by AI · not yet reviewed
How this idea evolved
No earlier ideas recorded for this concept yet.
Papers using this
- 2018Instagram hashtag pre-training
- 2018Instance discrimination
- 2018Do better ImageNet models transfer bette
- 2018decaNLP
- 2018Domain adaptive transfer
- 2019Adapters
- 2019Billion-scale semi-supervised
- 2019CPC v2
- 2019BoolQ
- 2019EfficientNet
- 2019LXMERT
- 2019Decoupled box proposals captioning
- 2019RLHF for LMs (Ziegler)
- 2019T5
- 2019BiT
- 2020Knowledge in LM parameters
- 2020GPT-3
- 2020BYOL
- 2020Learning to summarize from human feedbac
- 2020MMLU
- 2020ConVIRT
- 2020ViT
- 2020VL-BERT meta-analysis
- 2020DeiT
- 2021Prefix-Tuning
- 2021Switch Transformer
- 2021ALIGN
- 2021CLIP
- 2021Natural Instructions
- 2021ViT-VQGAN
- 2021Swin V2
- 2021Florence
- 2021Gopher
- 2021Fairseq MoE LMs
- 2022LaMDA
- 2022Megatron-Turing NLG
- 2022InstructGPT
- 2022Chinchilla
- 2022UniCL
- 2022OPT
- 2022CoCa
- 2022BEiT v2
- 2022EVA