Big Transfer recipe

How this idea evolved

Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that BiT cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.

• Domain-adaptive pre-training dataDomain adaptive transfer · 2018cited directly

  Reweight pre-training data towards the target domain; more data is not always better.

• Noisy Student trainingNoisy Student · 2019cited directly

  Self-training with an equal-or-larger student and noise added to it, iterated.

• Teacher–student self-trainingBillion-scale semi-supervised · 2019cited directly

  A teacher labels a huge unlabelled pool; a student learns from those pseudo-labels.