Perceiver AR: general-purpose, long-context autoregressive era

Over the previous couple of years, autoregressive Transformers have introduced a gentle stream of breakthroughs in generative modeling. These fashions generate every aspect of a pattern – the pixels of a picture, the characters of a textual content (usually in “token” chunks), the samples of an audio waveform, and so forth – by predicting one aspect after the opposite. When predicting the subsequent aspect, the mannequin can look again at those who had been created earlier.

Nonetheless, every of a Transformer’s layers grows dearer as extra parts are used as enter, and practitioners can solely afford to coach deep Transformers on sequences not more than about 2,048 parts in size. And so, most Transformer-based fashions ignore all parts past the newest previous (round 1,500 phrases or 1/6 of a small picture) when making a prediction.

In distinction, our not too long ago developed Perceiver models give wonderful outcomes on quite a lot of real-world duties with as much as round 100,000 parts. Perceivers use cross-attention to encode inputs right into a latent house, decoupling the enter’s compute necessities from mannequin depth. Perceivers additionally spend a set value, no matter enter measurement, at practically each layer.

Whereas latent-space encoding handles all parts in a single go, autoregressive era assumes processing occurs one aspect at a time. To deal with this drawback, Perceiver AR proposes a easy resolution: align the latents one after the other with the ultimate parts of the enter, and punctiliously masks the enter so latents see solely earlier parts.

Perceiver AR maps an enter sequence (P e r c e i v e r A R) to a small latent house by cross-attention to provide one latent for every goal token (3 latents proven, one for the targets A R <EOS>, for End Of Sequence). These latents are then processed by a deep stack of self-attention layers. Perceiver AR will be skilled for end-to-end autoregressive era, all whereas making use of very lengthy enter sequences.

The result’s an structure (proven above) that attends to as a lot as 50x longer inputs as customary Transformers, whereas deploying as broadly (and basically as simply) as customary decoder-only Transformers.

As context size or mannequin measurement will increase, the quantity of compute wanted to coach a mannequin grows. We are able to quantify the compute price range for various fashions by measuring their velocity on actual {hardware} (steps per second on TPUv3), because the enter context size and mannequin measurement improve. Not like different generative fashions like Transformer or Transformer-XL, Perceiver AR decouples enter context size from mannequin depth, permitting us to simply deploy the deep fashions wanted to mannequin lengthy sequences on current-generation TPUs or GPUs.

Perceiver AR scales significantly higher with measurement than each customary Transformers and Transformer-XL fashions at a variety of sequence lengths in actual phrases. This property permits us to construct very efficient long-context fashions. For instance, we discover {that a} 60-layer Perceiver AR with context size 8192 outperforms a 42-layer Transformer-XL on a book-length era process, whereas operating sooner in actual wall-clock phrases.

On customary, long-context picture (ImageNet 64×64), language (PG-19), and music (MAESTRO) era benchmarks, Perceiver AR produces state-of-the-art outcomes. Rising enter context by decoupling enter measurement from compute price range results in a number of intriguing outcomes:

Compute price range will be tailored at eval time, permitting us to spend much less and easily degrade high quality or to spend extra for improved era.
A bigger context permits Perceiver AR to outperform Transformer-XL, even when spending the identical on compute. We discover that better context results in improved mannequin efficiency even at reasonably priced scale (~1B parameters).
Perceiver AR’s pattern high quality reveals a lot much less sensitivity to the order wherein it generates parts. This makes Perceiver AR simple to use to settings that don’t have a pure left-to-right ordering, resembling knowledge like photos, with construction that spans multiple dimension.

Utilizing a dataset of piano music, we skilled Perceiver AR to generate new items of music from scratch. As a result of every new notice is predicted based mostly on the complete sequence of notes that got here earlier than, Perceiver AR is ready to produce items with a excessive stage of melodic, harmonic, and rhythmic coherence: