Attention Mechanism After 7 Years: Someone Finally Changed Its Underlying Logic

All large models are built on the same assumption: attention mechanism is correct.

The Transformer architecture has dominated for 7 years, with MHA (Multi-Head Attention) as its heart. The three matrices Q, K, V compute attention scores, and the KV cache stores historical context. The longer the generation, the larger the cache—this was a natural physical constraint that no one questioned.

Until Baidu turned the KV cache into a spillway in Unlimited OCR.

KV Cache: The Reservoir Problem of Large Models

First, let's understand the memory model of standard attention mechanism.

When generating the first token, the KV cache contains only 1 key-value pair. When generating the 1000th token, there are 1000 pairs. At the 100,000th token, 100,000 pairs. Linear growth. This is like a reservoir with only an inlet and no outlet—the water level only rises.

For chat scenarios, this is not a big problem. A conversation with a few thousand tokens doesn't take much GPU memory. But OCR is different—a 40-page document can easily output tens of thousands or even hundreds of thousands of tokens. The KV cache snowballs, causing memory to explode first, then speed crashes.

So all OCR models chose the same fallback: page-by-page processing. Cut long tasks into short ones, infer each page independently, with no memory between pages.

It works. But it's like breaking a novel into 40 unrelated summaries—information is lost between pages.

R-SWA: Installing a Spillway for the Reservoir

Baidu's R-SWA (Reference Sliding Window Attention) can be summarized in one sentence: install a fixed-capacity spillway for the KV cache.

Each time a new token is generated, the oldest token is evicted from the queue. The queue length is always constant. The KV cache occupancy is exactly the same whether outputting 10,000 tokens or 100,000 tokens.

But there is a subtle distinction—not all tokens are treated equally.

R-SWA divides tokens into two categories: reference tokens (visual tokens of the image + prompt) and output tokens (generated text tokens). For reference tokens, full visibility is always maintained—the original text is never lost. For output tokens, only the most recent 128 are kept—forget what should be forgotten.

This corresponds to a very intuitive cognitive model: when you are copying a book, your eyes can always see the original book (reference tokens fully visible), but you only focus on the last few lines of what you have written (output token sliding window). Earlier written content gradually fades from cognition, but the original book remains in view.

In implementation, Unlimited OCR replaces all attention layers with R-SWA. Not just a few layers, but all. This is a radical choice—meaning the model follows the principle of "original text fully visible, recent output reviewable, distant output forgotten" in every inference layer, consistently from bottom to top.

Why Hasn't Anyone Used This Before?

Good question. I didn't understand at first either—if sliding window is so natural, why hasn't anyone done this in 7 years?

The answer lies in the uniqueness of the OCR task.

OCR is essentially a transcription task, not a reasoning task. The model does not need to "remember" the content of page 3 to process page 30—it only needs to always "see" the original text. The distant history on the output side contributes little to the current generation because the output order of OCR strictly follows the reading order of the image.

In other words: OCR is a perfect scenario for R-SWA. The distant dependency of output tokens is very weak, while the full dependency of reference tokens is very strong—this is exactly the condition where sliding window attention can maximize its advantage.

But in general language modeling, distant dependency is indispensable. A foreshadowing planted at the beginning of a story is only revealed at the end. If the sliding window discards the beginning, the model loses context.

Therefore, the sentence at the end of the paper is especially thought-provoking: "R-SWA is a general parsing mechanism, and OCR is just the first stop."

If R-SWA can also work in ASR (Automatic Speech Recognition) and translation—transcription tasks naturally have the characteristics of "original text fully visible, output order dependency weak"—then its applicability far exceeds OCR.

DeepEncoder: Another Role Not to Be Ignored

R-SWA solves the constant memory problem, but one link is still missing: the number of visual tokens.

If a 1024×1024 PDF page is encoded into thousands of visual tokens, then "reference tokens fully visible" becomes empty talk—full visibility means thousands of tokens participate in every attention calculation, and the complexity O(n) still grows with the number of pages.

DeepEncoder fills the key gap here. It compresses a 1024×1024 page to only 256 visual tokens, a 16x compression ratio. More importantly, because visual tokens do not participate in state transitions under R-SWA—they always reside in the "reference zone" and are never evicted from the queue—no matter how long the document or how many pages, each page incurs only a fixed overhead of 256 tokens.

20 pages = 20 × 256 = 5120 visual tokens. In R-SWA's reference zone, the attention computation of 5120 tokens is completely tolerable. 40 pages? 10240 tokens. Still manageable.

This explains why Unlimited OCR can read dozens of pages in one inference within the standard 32K context—R-SWA controls memory expansion on the output side, DeepEncoder controls token count on the input side, and their combination completely eliminates the memory bottleneck of long document parsing.

Benchmarks: Not Just Beautiful Theory, Actually Usable

On OmniDocBench v1.5, Unlimited OCR achieves 93.23% vs DeepSeek OCR's 87.01%, a gap of 6.22 percentage points. This is a crushing overall score.

But more convincing is the efficiency data. In Flash Attention v3 latency tests:

• DeepSeek OCR's standard MHA: As decoding steps increase, time per step steadily climbs. A diagonal line going up.
• Unlimited OCR's R-SWA: A horizontal line from start to finish. Absolutely flat.

When outputting 6144 tokens, Unlimited OCR's TPS is 7847, DeepSeek OCR has dropped to 5822. Gap of 35%.

20-page document edit distance: 0.057. Above 40 pages: 0.11. Distinct-35 as high as 97%—almost no repeated output.

Formula CDM from 83.37 to 92.61. Table TEDS from 84.97 to 90.93. Text edit distance from 0.073 to 0.038.

A small MoE model with 500M activated parameters, only continued training for 4000 steps on top of DeepSeek OCR. The paper calls R-SWA a "free lunch" for parsing tasks—training cost almost unchanged, inference speed faster, constant memory, and overall performance improvement.

I was initially skeptical of the "free lunch" claim. But after seeing the data, I have to admit: for transcription tasks, R-SWA is indeed close to free.

The Person Signed as YY

Three core contributors of the paper: Youyang Yin, Huanhuan Liu* (project leader), YY† (technical director).

Two real names, the technical director with two-letter initials. In the GitHub acknowledgments, Deepseek-OCR and Deepseek-OCR-2 are listed first.

The three core authors of DeepSeek OCR: Wei Haoran, Sun Yaofeng, Li Yukun. In the V4 report this April, Wei Haoran had an asterisk—indicating he had left. The only one among the three to have publicly left.

Wei Haoran's career: started at Jieyue Xingchen → led GOT-OCR2.0 → built the entire OCR pipeline at DeepSeek (DeepEncoder, MoE decoder, from generation 1 to 2) → now appears in Baidu's paper, signed as YY.

From GOT-OCR2.0's end-to-end parsing, to DeepSeek-OCR's visual compression, to R-SWA's soft-forgetting attention. Same person, three places, an unbroken technical thread. He wasn't changing jobs; he was pursuing the ultimate solution to the same problem.

The paper's outlook mentions: next step, train to 128K context, build a prefill pool to let the model learn automatic page turning.

R-SWA is not the end of OCR. It is a new starting point for long-range parsing tasks—and the person at this starting point has been on this path for three years.

GitHub: https://github.com/baidu/Unlimited-OCR

HuggingFace: https://huggingface.co/baidu/Unlimited-OCR