The One Idea

Eight people, one paper, a Beatles joke

In June 2017, eight researchers working at Google posted a paper to arXiv. The title borrowed from the Beatles. The author list came with a footnote that has aged into a small legend: equal contribution, listing order is random.

Their ambition was modest by today’s standards. Translate better. Train faster. On the standard English-to-German benchmark they beat every previous result, ensembles included, and their largest model trained in three and a half days on eight graphics cards.

Nobody in that group was trying to build a chatbot. That is the first lesson worth keeping. The ideas that change everything rarely announce themselves. They show up as a better way to do something boring.

Before: reading through a keyhole

Before 2017, language models read the way you would read through a keyhole. One word, then the next, then the next. These were recurrent neural networks, and the idea was elegant: carry a running memory forward, update it with each word, and hope the important parts survive the trip.

Often they did not. By the end of a long sentence, the beginning had faded. Ask one of these models what “it” referred to twenty words back and it was guessing from a blurred summary.

Worse, it was slow in a way no budget could fix. Word nine had to wait for word eight. Word eight had to wait for word seven. You could buy a thousand processors and most of them would sit idle, waiting in line.

The problem was never intelligence. It was the queue.

The one idea: everyone listens to everyone

The Transformer threw the queue away. Instead of passing a memory from word to word, it lets every word look directly at every other word in the sentence, at the same moment, and decide how much each one matters to it.

Take the sentence: the animal didn’t cross the street because it was too tired. You know instantly that “it” is the animal. Change one word, too wide, and “it” becomes the street. Nothing about the word “it” changed. What changed is what it pays attention to.

Mechanically, each word asks a question, every other word holds up a label saying what it is about, and the strongest matches pass their meaning along. Engineers call these queries, keys and values. You can think of a room where everyone listens to everyone, and turns towards whoever is most relevant.

Two refinements made it work. The model runs several of these listening passes side by side, eight in the original design, so one can follow grammar while another follows who is who. And because everything now happens at once, each word carries a small stamp of its position, so the model still knows that “dog bites man” is not “man bites dog”.

The real breakthrough was not attention

Here is the flip. The title says attention is all you need. The history says something else. Attention already existed before 2017, bolted onto the old word-by-word models. What the Transformer really did was remove the waiting.

When every word is processed at the same time, the work stops looking like a queue and starts looking like a spreadsheet: large grids of numbers multiplied together. That is exactly what graphics chips were built to do. Suddenly, more hardware meant more model.

That is the unlock the last eight years were built on. Not a smarter algorithm. A shape of computation that finally fitted the machines. Scale did the rest.

Attention made the Transformer accurate. Parallelism made it inevitable.

From translation to your pocket

A year later, OpenAI trained a stack of Transformer layers on a large pile of books and called the result a Generative Pre-trained Transformer. GPT. Google went the other way with BERT, a Transformer that reads in both directions at once, and within two years it was helping Search understand what people actually meant.

Then the idea escaped language. The same architecture now sits underneath the assistant you asked something this morning, the translation in your browser, the code suggestions in an engineer’s editor, and many of today’s image and video generators.

So when the AI in your pocket follows a conversation, summarises a contract or rewrites an email in your tone, the honest explanation is not magic. It is an enormous number of words, each paying attention to all the others.

What the paper quietly traded away

Every winning design gives something up. The Transformer’s price hides in plain sight: if every word looks at every other word, the work grows with the square of the length. Double the text, and the attention work quadruples.

That one trade-off explains a lot of the AI you already feel. Why context windows have limits. Why long documents cost more. Why a large part of current research, from sparse attention to state-space models, is trying to keep the magic while escaping the square.

The authors know this better than anyone. All eight eventually left Google to build companies and labs of their own. Reunited on stage in 2024, Aidan Gomez said the field needs successors that can carry it to a new plateau of performance. The people who built the idea are among the most eager to replace it.