Running the Same Weights 64 Times Outperforms Models with Double the Parameters — Recurrent Transformer is Revolutionizing AI Reasoning

770M parameters, beating a 1.3B model.

Not through distillation, not through pruning, not any optimization trick you've heard of. Just running the same set of weights in a loop 16 times.

This project is called OpenMythos, with 14.6k stars on GitHub, by author Kye Gomez, 22 years old. Using pure PyTorch, starting from publicly available papers, he theoretically reconstructed Anthropic's most mysterious Claude Mythos architecture.

Open-sourced under the MIT license. Anyone can use it.

A Phenomenon Anthropic Won't Explain

Let's start with the background.

Claude's Mythos version exhibited a qualitative leap in reasoning ability—it can perform multi-step reasoning without outputting any intermediate steps. No Chain-of-Thought, no visible thinking process, the answer pops out directly, and it's correct.

Traditional Transformers can't explain this. Models like GPT, LLaMA have reasoning depth equal to the number of network layers; you can't add layers at inference time. Want more depth? You have to output more tokens to "show" the thought process.

But Mythos doesn't need that. It silently completes reasoning somewhere invisible.

Anthropic has not disclosed architecture details. Project Glasswing was only available to 50 institutions for internal testing. What's inside the black box of Mythos? Outsiders can only guess.

Kye Gomez decided to figure it out himself.

The Answer: Recurrent-Depth Transformer

The core hypothesis given by OpenMythos: Claude Mythos is built on a Recurrent-Depth Transformer.

The principle is not complicated, but counterintuitive.

A traditional Transformer is like a 100-story building, each floor decorated differently; data walks from floor 1 to floor 100, passing each floor once. Parameter count is tied to the number of layers.

A Recurrent Transformer is like a 6-story building, but the data loops 16 times inside the building. Same set of weights, same attention mechanism, same FFN, repeatedly processing the same data. Each loop, the hidden state goes deeper.

770M parameters × 16 loops ≈ inference depth of a 1.3B parameter model.

Key difference: Parameters don't increase, inference depth is adjustable.

Three-Stage Architecture: Prelude, Recurrent Block, Coda

How is it implemented? Three-stage structure:

Stage 1: Prelude – Standard Transformer layers, run once, encoding input tokens into hidden states and an input injection vector e.

Stage 2: Recurrent Block – The core. The hidden state h is updated each loop:

h_{t+1} = A·h_t + B·e + Transformer(h_t, e)

Note the e. The input injection vector participates every loop, preventing the hidden state from "drifting" during repeated cycling—forgetting what it's reasoning about.

Stage 3: Coda – Standard Transformer layers, run once, decoding the post-recurrent hidden states into output logits.

In, loop, out. That simple?

Of course not. Hidden beneath the simplicity are six ingenious innovation modules, each solving a fatal problem of recurrent architectures.

Six Life-Saving Inventions

① LTI Stabilized Injection – Prevent Training Explosion

What frightens recurrent models the most? Unbounded growth of hidden states across loops. Residuals accumulate, training diverges suddenly, loss spikes to the sky.

OpenMythos's solution is extremely elegant: model the recurrence as a discrete linear time-invariant system, guaranteeing spectral radius ρ(A) < 1 through mathematical construction. No matter how high the learning rate or how large the batch noise, the spectral radius of matrix A is always less than 1. Training won't explode.

One line of core code:

def get_A(self):
    return torch.exp(-torch.exp((self.log_dt + self.log_A).clamp(-20, 20)))

exp(-exp(x)) is always in (0, 1). Mathematics guarantees stability, not tuning.

② Adaptive Computation Time (ACT) – Prevent Overthinking

More loops are always better? Wrong. Beyond a certain depth, excessive cycling reduces prediction quality.

Solution: Each position learns a halting probability; when cumulative probability exceeds 0.99, updates stop. Simple tokens stop early, difficult tokens compute more. Within the same batch, different tokens can exit at different depths.

Bonus theoretical insight: Under certain assumptions, the ACT mechanism makes the model Turing complete.

③ Loop Index Embedding – Make Each Loop Different

Running the same weights 16 times, doing the same thing each time? That would make loops meaningless.

Solution: Similar to RoPE encoding sequence positions, inject sinusoidal position signals for loop depth. The same attention layer receives different position signals at the 3rd loop and the 12th loop, thus performing different operations.

④ Depth LoRA Adapter – Balancing Parameter Sharing and Expressiveness

Pure weight tying lacks expressiveness; completely different weights offer no parameter savings.

Solution: Share large weight matrices + a rank-r small adapter for each loop depth. Total parameter overhead is minimal, but each loop's behavior is subtly different.

⑤ Dual Attention Mechanism – GQA / MLA Switchable

• GQA: Fewer KV heads than Q heads, reducing KV cache, supporting Flash Attention 2
• MLA (DeepSeek-V2 style): Cache compressed KV latent variables, memory reduced 10-20 times

Use MLA in production, GQA for training; switch with one line of configuration.

⑥ Fine-Grained MoE – Breadth Coverage

The recurrent architecture explains reasoning depth, but what about domain breadth?

64-512 small SwiGLU FFN units as routed experts, each token activates top-K (about 5% activation rate). Plus always-activated shared experts to absorb cross-domain general knowledge.

The most subtle point: As the hidden state evolves through loops, the router may select different expert subsets at different depths. Within a single forward pass, each loop is computationally unique.

Counterintuitive Data

For variants above 100B, the context window jumps to 1M tokens, output 128k. Loop depth ranges from 16 to 64, and can be further increased at inference—training with 16 loops, running 32 loops at inference still improves performance. This is called Depth Extrapolation, something traditional Transformers cannot do.

Train with 6 layers of recurrence, test with 10 layers and still answer correctly. GPT cannot do that.

Three Superpowers Exclusive to Recurrent Architectures

Superpower 1: Continuous Latent Space Reasoning > Discrete Token Reasoning

Chain-of-Thought can only choose one direction per step, similar to depth-first search. The Recurrent Transformer operates in continuous latent space, can encode multiple alternative next steps simultaneously—similar to breadth-first search. Not going down a single path blindly, but exploring multiple paths in parallel.

Superpower 2: Three-Phase Grokking

The systematic generalization of Recurrent Transformers is not gradual, but emerges in three phase transitions: Memorization → In-Distribution Generalization → Systematic Generalization. The third phase emerges suddenly and sharply.

This explains why Mythos feels "qualitatively different" on novel problems—ability enters via phase transition, not gradual emergence.

Superpower 3: Scalable at Inference

Traditional models are fixed after training. Recurrent models can dynamically adjust computation at inference—simple problems use fewer loops, complex problems use more loops. Same model, same batch, different tokens, computation adaptively allocated.

Run with Five Lines of Code

from open_mythos import MythosModel, MythosConfig

config = MythosConfig(dim=2048, num_experts=64, loop_iters=16)
model = MythosModel(config)

output = model(input_ids)  # Single forward pass, internally loops 16 times

pip install open-mythos, run the 1B variant locally. No need for multiple GPUs, no distributed setup.

Let's Be Realistic

Let me clarify a few things.

OpenMythos is a theoretical reconstruction, not a code leak. It is derived from publicly available academic papers, and may differ from Anthropic's actual implementation.

There are no pre-trained weights currently. You can run the architecture, but you can't use it directly—you have to train it yourself. The 3B training script is written with PyTorch FSDP, the dataset is FineWeb-Edu, but training costs are not low.

There is no affiliation with Anthropic. The project's statement is clear.

What Does the Recurrent Architecture Really Mean?

The ceiling of traditional Transformers is obvious: stacking layers → parameter explosion → inference cost out of control → rely on CoT as a crutch.

The recurrent architecture takes a different path: don't stack parameters, stack computation. Run the same weights a few more times, freely expand inference depth. Parameter efficiency and reasoning ability are no longer bound together.

770M matching 1.3B is not the end. 1T parameters × 64 loops yields an inference depth equivalent to a 4000+ layer traditional Transformer—no GPU can run 4000 layers, but the recurrent architecture only occupies 1T of memory.

OpenMythos is a blueprint. It proves one thing: A leap in reasoning ability does not require a leap in parameters.

Who says models have to keep getting bigger?

Project link: https://github.com/kyegomez/OpenMythos

Related papers:

• Parcae: Predictable Scaling Laws for Recurrent Language Models (Prairie et al., 2026)
• Loop, Think, & Generalize (arXiv 2604.07822)
• Reasoning with Latent Thoughts (arXiv 2502.17416)
• Universal Transformers (Dehghani et al., 2018)