Open-weights LLMs

How this idea evolved

Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that OPT cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.

• In-context / few-shot learningGPT-3 · 2020cited directly

  A large model solves a new task from a few examples in its prompt, with no gradient updates.

• Knowledge-grounded dialogue LMsLaMDA · 2022cited directly

  Fine-tune dialogue models to consult external tools and knowledge for factual answers.

• Compute-optimal trainingChinchilla · 2022cited directly

  For a fixed budget, grow parameters and training tokens together; most big LMs were undertrained.