How far memorization can go

This is part 3 of the four article series "AI Is (Mostly) Memorization":

1. AI Is (Mostly) Memorization
2. Why memorization works
3. How far memorization can go - this article
4. Where memorization ends and real innovation begins

In the previous articles, I argued that modern AI is fundamentally a memorization system.

This claim is easy to accept for obvious pattern-matching tasks. Recommendation engines remember preferences. Face recognition systems remember visual patterns. Language models remember text, code and documented reasoning traces.

But what about harder problems like writing sophisticated software, solving challenging mathematics problems, assisting scientific research, or producing seemingly creative work. Would those require something beyond memorization?

The surprising answer is that memorization can go much further than we might expect.

Most "new" problems aren't actually new

When we encounter a difficult task, we often focus on what is novel about it. But novelty is usually only a small fraction of the problem.

When software engineers get feature requests they've never seen all the time. What do they do? They usually start from recalling prior tasks, consider existing modules, known libraries and standard design patterns. They normally implement the "new" feature by fitting old knowledge to the new spec.

The same story happens everywhere. A tricky math problem looks original but uses familiar techniques in a new arrangement. A lawyer handling a novel case still argues using the same structures that have worked before. A doctor seeing unusual symptoms sometimes still relies on patterns from cases they've seen or studied.

Why does this approach work? Because hard problems are not always hard in every dimension. Break them into pieces, and each piece usually lands somewhere familiar. The novelty is often in the combination, not the components.

This is what memorization systems are great at:

1. Decompose a problem into smaller, better manageable pieces.
2. Match each piece to relevant prior data points or patterns.
3. Explore variations near those matches and adapt those patterns.
4. Compose partial solutions of small problems into a coherent whole solution.

Consider asking an AI to build a restaurant reservation software system. At first glance, this appears to be a complex new task. After decomposition, it becomes a collection of familiar ones:

• User accounts
• Calendar interfaces
• Database design
• Email/SMS notifications
• Reporting dashboards
• ...

AI rarely invents these pieces from scratch. Instead, it adapts and combines solutions they already know. Each of such components can be further broken down into submodules, and even lines of code level where familiarity can be found.

Much of what we call novelty is really recombination under new constraints. Decomposition plus memorization is surprisingly effective for a wide range of tasks.

The industry keeps pushing this approach further, for both classical machine learning tasks and LLMs. We’ve been seeing lots of research and best practices on how to collect or synthesize larger datasets, how to enhance label quality, how to extend the context window size, and improve retrieval systems. People developed specific memory technologies and systems for agents to remember better.

Most of the recent progress comes from expanding what models can remember and how well they can access it. Each of these advances stretches the reach of memorization a little further, which is part of why it keeps surprising us how much ground it can cover.

The spectrum of tasks where AI fits in

Now let's have an honest view of where AI sits across different kinds of tasks. Think of it as a gradient from tasks where memorization is almost fully sufficient to tasks where it increasingly falls short. As we move along that gradient, the human role changes too.

Retrieval. Recalling facts, specifications, prior cases, and reference knowledge. A doctor remembering drug interactions. A lawyer citing relevant case law. An engineer looking up material specifications. 

This is where AI is most reliable. Humans forget and have limited memory capacity. AI doesn't have those limits, though it still hallucinates. Humans remain essential as verifiers.

Recombination. Assembling known components into expected configurations. Formatting documents, translating between languages, generating boilerplate code, classifying images, recommending products. 

When training data is rich enough that pattern-matching covers most cases, AI often matches or exceeds human performance. The human role shifts towards checking quality with domain expertise.

Adaptation. Applying known patterns that do not quite fit the template. Debugging complex software systems, diagnosing unusual medical cases, designing experiments, building new product features using existing components. 

This requires judgment about which patterns apply, how to modify them, and when to abandon them. AI becomes a powerful collaborator by surfacing options and speeding up exploration, but humans still own the direction and accountability.

A lot of professional work sits in these categories, including some highly paid work. Much of what knowledge workers do every day is really retrieval, recombination, or adaptation. These activities are valuable and essential. But they all share one thing: they work with what's already been seen before.

So the natural question becomes: what happens when memorization and adaptation aren’t enough?

That's the ceiling of memorization and real innovation begins.