After Transformer, What Exactly Is AI Improving?

I recently came across an interesting meme. Its basic idea was:

It is a meme, of course, but it is funny because it captures something real about AI's development over the past few years:

As a comment on the backbone architecture of large language models, the meme is remarkably accurate. As evidence that AI has no new foundational technology, however, it goes too far.

The central point is this: Since Transformer appeared in 2017, there truly has not been another dominant architecture replacement on the scale of “RNN/CNN → Transformer.”

What Are Transformer and Self-Attention?

Transformer is a neural network architecture. Self-attention, one of its core mechanisms, lets a model directly evaluate relationships among different positions in a sequence. Most mainstream large language models are still built on the Transformer family.

Frontier large language models since then can be understood roughly as:

Transformer

• More and better data
• More compute
• Better training methods
• More efficient model structures
• More inference-time compute
• Tool use

It is not “Transformer → a completely different theory → the next generation of AI.”

What the Meme Gets Right

Looking back at machine learning around the year 2000 reveals many radically different technical approaches:

• SVM / Kernel Method
• Gaussian Process
• Bayesian Network
• HMM
• Decision Tree
• Boosting
• Graphical Model
• Neural Network

Their modeling assumptions, mathematical formulations, and even training methods could be entirely different.

Look at many public large models today, however, and their backbones appear strikingly similar.

DeepSeek-V3, for example, contains many innovations but is still built within the Transformer family. Its main changes can be summarized as:

Transformer + MoE + MLA + Multi-Token Prediction + training-engineering optimization

MoE (Mixture of Experts)

Think of a model as containing many expert modules while invoking only a subset for each token. The model can be very large without evaluating all of its parameters every time.

MLA (Multi-head Latent Attention)

An attention improvement used by DeepSeek whose main goal is to compress the intermediate state retained during inference, reducing memory and compute costs.

A map of the technical path after 2017 therefore looks more like this:

Transformer
    │
    ├── Scaling
    ├── MoE
    ├── RoPE / GQA / MLA
    ├── FlashAttention
    ├── Better Data
    ├── RLHF / DPO / RL
    ├── Long Context
    ├── Test-time Compute
    └── Tool Use

That is closer to reality than assuming that an entirely new backbone architecture appeared in each of 2020, 2023, and 2026.

Scale Itself May Be One of This AI Wave's Biggest Discoveries

This point is often underestimated.

In other words, You do not necessarily need to invent a new algorithm first. Scaling an existing method in the right proportions may be enough for performance to keep improving predictably. At the same time, post-training methods have advanced substantially.

Scaling Laws

The study of relationships among model size, training data, compute, and model performance. Its central lesson is not simply that bigger is better, but that the relationship between these investments and the resulting performance follows estimable patterns rather than pure trial and error.

Chinchilla later showed that many large models did not lack parameters; they lacked enough training data.

The emphasis gradually shifted from “bigger models are better” to properly balancing model size, data volume, and training compute.

That is deeply counterintuitive.

The more familiar route used to be:

Invent a smarter algorithm → improve performance

The era of large language models introduced another route:

Find a sufficiently general architecture → scale data and compute by orders of magnitude → capabilities keep growing

That is why the meme's it worked somehow may be a joke, but it captures a genuinely unusual phenomenon.

“Just Make Transformer Bigger” Is Not Quite Right Either

Many important techniques have appeared over the past few years; they simply did not replace the backbone architecture.

FlashAttention

A more efficient way to compute attention. It reduces data movement between GPU memory and cache, improving speed and lowering memory usage.

It is closer to computing the same formula more intelligently than to inventing a new formula.

It does not change the mathematical result that attention computes. Instead, it redesigns how the GPU reads data and performs that computation.

In simple terms:

Mathematics: Attention is still Attention
Implementation: redesign how the computation moves and processes data

These techniques matter, but they are low-level optimization at the computer-systems layer, not an architecture replacement at the level of machine-learning theory.

MoE is a similar case.

A standard Transformer:

token
  ↓
entire FFN

MoE:

token
  ↓
Router
 ├→ Expert 3
 └→ Expert 17

This makes configurations such as “600B total parameters, 30B active” possible: the model can have enormous total capacity without running every parameter for every token.

If the test is whether the basic paradigm has changed, these techniques are closer to a major modification of Transformer, rather than a new foundational AI paradigm.

It is therefore plainly wrong to say that AI has seen no innovation in recent years.

A more accurate way to put it is:

Innovation has been abundant, but for a long time there has been no architectural revolution capable of displacing Transformer as the backbone.

The Crucial Change: Scaling at Inference Time

After 2024, one trend became increasingly clear: scaling no longer happens only during training.

The old path was “more GPUs → train a larger model → get a stronger model.” A second path now exists: “the same model → spend longer computing on the answer → perform more search, verification, and deduction → produce a better answer.”

Test-time Compute / Inference-time Compute

The amount of compute a trained model spends while answering an actual question.

The earlier emphasis was usually on training a larger model. The new question is whether the same model can spend more time computing when it faces a difficult problem.

Scaling can now be understood across four layers: parameter count, training data, training compute, and inference-time compute.

This shift may matter more than simply growing a model from 70B to 700B parameters.

But if the question is, “Does this count as a new foundational theory of neural networks?” the answer remains: Not really.

It is better understood as a change in training methods and in how models are used, not as a reinvention of the neural network's basic structure.

Some Researchers Really Are Trying to Replace Transformer

State Space Model / Mamba

A family of sequence models distinct from standard Transformer attention. It preserves and passes information through an internal state, offering the important advantage of lower compute costs on very long sequences.

This is not a small tweak to attention; it genuinely uses a different way to process sequences:

Transformer: token ↔ token through Attention
Mamba: token → hidden state → hidden state

One of Mamba's main goals is to reduce the cost that attention incurs on long sequences, allowing compute to grow roughly linearly with sequence length.

If the test is whether the model's underlying skeleton truly changes, Mamba comes much closer to genuine foundational architecture innovation.

The problem is that it has not yet displaced the industry's mainstream path in the way Transformer did after 2017.

For now, then, Transformer's dominance remains formidable.

How Accurate Is the Meme?

If we score the meme's claims:

A more precise description would be:

What has been remarkable about the past eight or nine years is not a lack of AI innovation, but the extraordinary durability of Transformer as a foundation. Many of the most important advances occurred in scale, data, training, inference, and computing systems rather than through repeated replacement of the neural network's basic architecture.

That leads to a more important question:

Are we approaching Transformer's ceiling, or have we accidentally found a general architecture that can serve us for many years?

We do not yet have an answer.

If another Transformer-scale breakthrough arrives, it may involve more than another adjustment to the attention formula.

A larger break may come from changing some of today's basic assumptions:

• Fixed-parameter models: Once training ends, a model's core parameters largely stop changing. Must future models still work this way?
• Context Window: Models now rely mainly on the context supplied with one request for temporary information. Will future systems have more natural, longer-lived memory?
• Token-by-token inference: Most language models still generate output one token at a time. Must future systems continue to work this way?

A genuine change to any one of these assumptions might be what finally produces another shift like the one in 2017.

When that happens, the “AI then / AI now” meme may finally need to be redrawn.