objectiveIntroduced by U-PaLM · 2022
Mixture-of-denoisers objective
Train on a mix of short-span, long-span and prefix denoising tasks.
Drafted by AI · not yet reviewed
How this idea evolved
Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that U-PaLM cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.
Replace random spans with sentinels and train the decoder to fill them in.
Hide some tokens and predict them from context on both sides.
Share weights across layers and factorise embeddings to shrink BERT.