evaluationIntroduced by Codex · 2021

Code LLMs & pass@k

LMs fine-tuned on code, measured by whether sampled programs pass unit tests.

Drafted by AI · not yet reviewed

How this idea evolved

Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that Codex cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.

In-context / few-shot learningGPT-3 · 2020cited directly
A large model solves a new task from a few examples in its prompt, with no gradient updates.
Knowledge stored in LM parametersKnowledge in LM parameters · 2020two citations away
Answer factual questions from model weights alone, with no retrieval.
Neural scaling lawsKaplan scaling laws · 2020two citations away
Loss falls as a power law in model size, data and compute.

Papers using this

2022PaLM