What Is an Attention Mechanism?

Models & Architecture Also known as: self-attention

An attention mechanism is the part of a modern AI model that lets it weigh which pieces of the input matter most for each part of the output. It is the core idea behind the transformer, the architecture that powers today's large language models.

How an attention mechanism works

Attention lets a model consider all the parts of an input at once and decide how much each one should influence a given step. When a model reads a sentence, attention is what lets it link a pronoun back to the noun it refers to, even if they are far apart, by assigning more weight to the relevant words. Earlier approaches read text strictly in order and struggled with those long-range links, which is the limitation attention removed.

Why attention mechanisms matter

Attention matters because it is what made large language models practical. By letting a model weigh relationships across an entire input in parallel, it enabled both the quality and the scale of training that current models depend on. Most of what modern AI can do with language traces back to this one idea, introduced in the 2017 paper that defined the transformer.

Frequently asked questions.

The stuff we hear most on the first call. Don't see yours? Book a 30-minute conversation.

What is self-attention?

Self-attention is attention applied within a single sequence, letting each part of the input weigh its relationship to every other part. It is the specific form used in transformers.

Why is the attention mechanism important?

It lets a model capture relationships across a whole input at once, which made today's large language models possible. The transformer architecture is built around it.

Want to put AI
to work?

We work with leadership teams to find the right opportunities, define the strategy, and build the systems that move the business forward.

Explore the Workshop Discuss a Custom AI Project