This is part 4 of the four article series "AI Is (Mostly) Memorization": AI is a memorization engine at its core. It retains historical patterns and solves problems through retrieval, recombination, and adaptation. This relies on one assumption: that the future will look enough like…
This is part 4 of the four article series "AI Is (Mostly) Memorization":
Where memorization ends and real innovation begins - this article
AI is a memorization engine at its core. It retains historical patterns and solves problems through retrieval, recombination, and adaptation.
This relies on one assumption: that the future will look enough like the past. Most of the time it does, and memorization is remarkably powerful. But when it doesn't, we hit a wall of model performance.
The real limits
AI doesn't struggle with hard problems. It struggles with structurally unfamiliar ones, where the stored patterns no longer map onto reality.
When the world shifts. During COVID, the travel recommendation system broke, because human travel behavior changed due to pandemic-driven visa constraints, quarantine policies, and fear of infection. The model has no historical pattern that matches the new situation.
The same happens with any cold start problems: a product launching in a new culture, a market reacting to a new event. These aren't bugs. They're the built-in limits of any system that mostly interpolates from what it's already seen.
When causation matters, not just correlation. Umbrella sales and traffic accidents both rise on rainy days. Without understanding that rain causes both, you might wrongly conclude that banning umbrellas would reduce accidents.
Many AI models mainly work on correlation and are therefore limited there. Recent reasoning LLMs seem better at tracing causal patterns, but they’re still limited to the reasoning patterns in the training data, and can't really go beyond them.
When you need to touch the real world. You can train a model on every published chemistry paper in history. It can effectively narrow the search space for drug candidates, but it still can't tell you whether a new drug candidate will work in a living human body. That requires running the experiment.
The same applies to robotics: AI trained in simulated environments still requires trial and error in the real world. Some signals can't be extracted from any dataset. They have to be generated through interaction with reality.
When goals are contested. Should a city optimize for growth, sustainability, or equity? Should an app optimize for profit or user wellbeing? These aren't prediction problems. They involve competing values, where pattern matching can’t tell you which value should win. That's a human question, not a data question.
These aren't edge cases. They describe some of the most important problems humans want to tankle: scientific breakthroughs, policy decisions, strategic pivots. When the situation is truly new, memorization helps less and less.
Beyond memorization: what humans add and how AI can help
Human intelligence also runs on memorization. Charles Darwin built his theory over decades of observation and reading across geology, economics and natural history. Issac Newton stood on the shoulders of giants. Every great innovator built on a large foundation of retained knowledge.
But humans can do something on top ofmemorization that current AI still can't. We question the frame by asking "what if we're solving the wrong problem?". We reason about causes, not just correlations. We make judgment calls when the data is incomplete. We connect ideas across different domains to create something genuinely new.
These capabilities all share one trait: they require stepping outside the patterns rather than following them. They require noticing when the map no longer matches the territory, and having the judgment to act on that mismatch.
That extra step is small in volume but huge in impact. It's the difference between optimizing a system and creating a new system.
AI can support this kind of work well. It compresses context, surfaces relevant evidence, and frees you up so you can focus on the thinking that matters.
What this means for you
If you've followed this series, you now have a mental model for AI that's more useful than "it's magic" or "it's just statistics." Here's what to do with it.
Hand over the memorization, not the thinking. Humans are mediocre memorizers. We forget relevant details, miss patterns in large datasets, and struggle to hold more than a few variables in our heads at once. AI doesn't have these problems. Use it for what it's genuinely better at: searching large solution spaces, surfacing relevant past cases, drafting first versions, catching patterns you'd miss. Then you do the part that actually matters: deciding what's right, what's missing, what doesn't fit, and what to do about it.
Take an honest look at your own work. How much of your day involves matching known patterns to familiar problems? Generating routine reports, replying common questions, writing test cases for code, sharing your past best practices? This type of work is right in AI’s sweet spot. You should leverage AI to accelerate the workflow and let it consolidate your knowledge and know-hows. This way you will be more efficient and more scalable.
Invest in the skills memorization can't replace: Learn to frame problems well by asking "are we even solving the right problem?" Think about causes, not just what the data shows. Get comfortable making decisions when the answer isn't obvious. Stay close to the real world and ground truth that models miss. Communicate with other people to learn the most from them. Build cross domain thinking and become a pattern breaker.
AI is the most powerful memorization system ever built. It compresses the past and deploys it at a scale no human can match. That alone is enough to transform industries and reshape how we work.
But the past isn't always a reliable guide.
The people who thrive alongside AI won't be the ones who try to out-memorize it. They'll be the ones who understand where memorization ends, know how to use AI well, and are willing to step into territory where no data, no pattern, and no prior reference can tell you what to do next.
That's exactly where the most meaningful work begins. And there's never been a better time to start than now.
This is part 3 of the four article series "AI Is (Mostly) Memorization": In the previous articles, I argued that modern AI is fundamentally a memorization system. This claim is easy to accept for obvious pattern-matching tasks. Recommendation engines remember preferences. Face recognition systems remember…
This is part 3 of the four article series "AI Is (Mostly) Memorization":
In the previous articles, I argued that modern AI is fundamentally a memorization system.
This claim is easy to accept for obvious pattern-matching tasks. Recommendation engines remember preferences. Face recognition systems remember visual patterns. Language models remember text, code and documented reasoning traces.
But what about harder problems like writing sophisticated software, solving challenging mathematics problems, assisting scientific research, or producing seemingly creative work. Would those require something beyond memorization?
The surprising answer is that memorization can go much further than we might expect.
Most "new" problems aren't actually new
When we encounter a difficult task, we often focus on what is novel about it. But novelty is usually only a small fraction of the problem.
When software engineers get feature requests they've never seen all the time. What do they do? They usually start from recalling prior tasks, consider existing modules, known libraries and standard design patterns. They normally implement the "new" feature by fitting old knowledge to the new spec.
The same story happens everywhere. A tricky math problem looks original but uses familiar techniques in a new arrangement. A lawyer handling a novel case still argues using the same structures that have worked before. A doctor seeing unusual symptoms sometimes still relies on patterns from cases they've seen or studied.
Why does this approach work? Because hard problems are not always hard in every dimension. Break them into pieces, and each piece usually lands somewhere familiar. The novelty is often in the combination, not the components.
This is what memorization systems are great at:
Decompose a problem into smaller, better manageable pieces.
Match each piece to relevant prior data points or patterns.
Explore variations near those matches and adapt those patterns.
Compose partial solutions of small problems into a coherent whole solution.
Consider asking an AI to build a restaurant reservation software system. At first glance, this appears to be a complex new task. After decomposition, it becomes a collection of familiar ones:
User accounts
Calendar interfaces
Database design
Email/SMS notifications
Reporting dashboards
...
AI rarely invents these pieces from scratch. Instead, it adapts and combines solutions they already know. Each of such components can be further broken down into submodules, and even lines of code level where familiarity can be found.
Much of what we call novelty is really recombination under new constraints. Decomposition plus memorization is surprisingly effective for a wide range of tasks.
The industry keeps pushing this approach further, for both classical machine learning tasks and LLMs. We’ve been seeing lots of research and best practices on how to collect or synthesize larger datasets, how to enhance label quality, how to extend the context window size, and improve retrieval systems. People developed specific memory technologies and systems for agents to remember better.
Most of the recent progress comes from expanding what models can remember and how well they can access it. Each of these advances stretches the reach of memorization a little further, which is part of why it keeps surprising us how much ground it can cover.
The spectrum of tasks where AI fits in
Now let's have an honest view of where AI sits across different kinds of tasks. Think of it as a gradient from tasks where memorization is almost fully sufficient to tasks where it increasingly falls short. As we move along that gradient, the human role changes too.
Retrieval. Recalling facts, specifications, prior cases, and reference knowledge. A doctor remembering drug interactions. A lawyer citing relevant case law. An engineer looking up material specifications.
This is where AI is most reliable. Humans forget and have limited memory capacity. AI doesn't have those limits, though it still hallucinates. Humans remain essential as verifiers.
Recombination. Assembling known components into expected configurations. Formatting documents, translating between languages, generating boilerplate code, classifying images, recommending products.
When training data is rich enough that pattern-matching covers most cases, AI often matches or exceeds human performance. The human role shifts towards checking quality with domain expertise.
Adaptation. Applying known patterns that do not quite fit the template. Debugging complex software systems, diagnosing unusual medical cases, designing experiments, building new product features using existing components.
This requires judgment about which patterns apply, how to modify them, and when to abandon them. AI becomes a powerful collaborator by surfacing options and speeding up exploration, but humans still own the direction and accountability.
A lot of professional work sits in these categories,including some highly paid work. Much of what knowledge workers do every day is really retrieval, recombination, or adaptation. These activities are valuable and essential. But they all share one thing: they work with what's already been seen before.
So the natural question becomes: what happens when memorization and adaptation aren’t enough?
That's the ceiling of memorization and real innovation begins.
This is part 2 of the four article series "AI Is (Mostly) Memorization": Modern AI systems are very good prediction engines at their core. A recommendation model predicts what you'll click next. A language model predicts the next words in a sentence. A vision model…
This is part 2 of the four article series "AI Is (Mostly) Memorization":
Modern AI systems are very good prediction engines at their core.
A recommendation model predicts what you'll click next. A language model predicts the next words in a sentence. A vision model predicts what object is in an image. A fraud detection model predicts whether a transaction is suspicious.
Machine learning works across so many domains because the world is not random, it has structure and repeating patterns.
These patterns largely come from three fundamental sources: physical laws, human nature, and social dynamics. Each creates regularities that make the past a useful guide to the future.
Physical laws: the foundation of consistency
At the most basic level, physical laws ensure that certain patterns repeat.
An apple fell from a tree yesterday, and it will fall tomorrow, accelerating at about ~9.8 m/s². Water freezes at around ~0 °C, whether it is in New York or Tokyo, in 1900 or 2026.
These laws shape everything we observe. Weather follows thermodynamic principles. Light reflects according to optical laws. Sound travels based on acoustic physics.
Consider pedestrian detection in a video surveillance system. It works not only because the AI model has seen many pedestrian images, but because human bodies have consistent structure, move within physical constraints, and are illuminated by light in predictable ways. The model memorizes these regularities, then applies them to recognize new cases.
Weather forecasting shows the same idea. The atmosphere is complex and chaotic, but it follows physical laws such as fluid dynamics and thermodynamics. Similar conditions often lead to similar outcomes. Machine learning models trained on large amounts of historical weather data can pick up these patterns and make accurate predictions.
Human nature: stable patterns in behavior (the persistence of psychology)
Humans are not random. We're shaped by biology, evolution, and cognitive limits.
We seek food, safety, and belonging.We avoid pain and prefer immediate reward. We pay attention to faces, movement, and novelty. We fall for the same cognitive biases, such as anchoring on first impressions, overreacting to recent events, and fearing losses more than we value gains.
These patterns have remained stable over long periods of time during the history of evolution.
This is why machine learning works so well for applications involving human behavior. Even though each person feels unique, the behavior often follows patterns, and the decisions fall along familiar lines. People who liked certain music or movie genres in the past tend to like similar ones in the future. Recommendation systems rely on this structure, and their predictive power is strong enough to generate billions in revenue.
This doesn’t mean humans are rigid or deterministic. It means that behavior has structure that makes it more predictable than it first appears.
Social dynamics: patterns at the collective level
When many individuals interact in social and economic systems, higher-order patterns appear at a larger scale.
Information spreads through social networks in recognizable ways. Communities develop habits and norms that stick around. Markets react to news in familiar patterns. These regularities emerge from the interaction of many partly predictable humans, creating repetition at the collective level.
Language is a good example. A language model learns that certain words follow others not because of physical law, but because millions of speakers have implicitly agreed to use language in coordinated ways. Grammar and vocabulary become social conventions, and those conventions create stable statistical structure. That structure can then be learned and predicted.
In areas like advertising and recommendations, we see how social proof and network effects create predictable chain reactions. Products that get early traction often get even more, following power-law distributions that appear across markets and time periods. In data mining across industries, I’ve seen models learn recurring seasonality patterns and business cycles.
Even in domains that feel highly chaotic, like trending topics, viral content, or meme culture on social media, there are underlying patterns in how information spreads, how attention builds and fades, and how communities form. These patterns are not deterministic, but they are far from random. They emerge from social network structure, the psychology of sharing, and the way people react to incentives.
Prediction is easier than understanding
AI does not need to understand why a pattern exists. It only needs the pattern to repeat.
A model doesn't need to understand fashion to predict what clothes will sell next season. It doesn't need to understand humor to predict which post will spread. It doesn't need to understand medicine to flag something unusual in a scan.
Memorizing the pattern is enough. Making a prediction is enough.
But how far can memorization really go? Can it handle genuinely hard problems, or only straightforward pattern matching?
The next artical will talk about: How far memorization can go.
The term Artificial Intelligence comes from a bold ambition: to build machines that can think, understand, and reason about the world the way humans do. Recently, that ambition feels very close. Recommendation systems seem to know exactly what we want before we do. Computer vision…
The term Artificial Intelligence comes from a bold ambition: to build machines that can think, understand, and reason about the world the way humans do.
Recently, that ambition feels very close. Recommendation systems seem to know exactly what we want before we do. Computer vision models detect faces, objects, and tumors more accurately than ever. Large language models write essays, generate code, summarize reports, and even support scientific research.
So where does this power actually come from? How does AI learn, reason, and solve problems?
After more than 15 years working in this field, my answer is simple: AI mostly memorizes.
I want to explain the source of power of AI and its inherent limitations in a four article series:
Most successful AI systems we see today are built on machine learning. The word “learning” implies understanding, reasoning, or discovering, like a student or a scientist figuring out how the world really works
In practice, most AI systems do something simpler:
They remember patterns from the past and reuse them when the present looks similar. That’s what learning usually means.
A face recognition system captures patterns of visual similarity between faces through vector computation. It memorizes how the same face changes under different lighting, angles, ages, and expressions.
A recommendation system tracks your behavior and compares with millions of others. Based on what you clicked, watched, bought, and ignored, then gives the closest recommendation, without really know who you are and what your taste is.
A large language model generates text by following statistical patterns: which words tend to follow others, what arguments commonly come next, how explanations are usually structured, and what kinds of responses have worked for similar questions.
In each case, the AI system works not because it understands the world, but because it has stored enough patterns from the past to produce a good answer now.
That is what I mean by memorization.
AI models that are built to memorize
Many of the most important models in machine learning are purposely built to store the past and look it up later. Here are three examples that span decades of AI research:
K-Nearest Neighbors (KNN), developed in the 1950s, is a simple and effective memorization engine. The model just stores all training examples. When a new input arrives, it finds the closest matches, retrieves the memorized labels, and generates the prediction through majority voting. It is the foundation of modern algorithms such as Collaborative filtering, and the the now popular Retrieval Augmented Generation (RAG) systems.
Linear regression is the simplest form or regression in statistics and machine learning. Linear regression or regression for more complex functions is still a compact form of memorization. Instead of remembering every example, it compresses training data into a small set of parameters. Those parameters capture relationships that repeatedly appeared in the past. "Learning" becomes the process of adjusting those parameters to make the memory accurately.
Transformers, the architecture behind modern large language models, memorizes at two levels. First, the feed-forward layers contain most of the model's parameters and serve as long-term memory. Factual knowledge and reasoning patterns are encoded into these weights during training, allowing the model to retrieve during inference. Second, the attention mechanism acts as short-term, dynamic memory by selecting the most relevant information from the current context. This helps the model identify the most important tokens when generating the next output.
From the simplest classifier to the largest language model, the underlying recipe is the same: store information from the past in some reusable form, then retrieve and combine it when needed. What changes across models is how the past is stored and how flexibly it can be reused.
A broader view of memorization
The word memorization might sound narrow, like recalling a poem or a phone number.
But in real life, human expertise depends heavily on it. A software engineer builds up intuition about system behaviors, failure modes, and debugging strategies across many projects. A physician recognizes patterns from years of patient cases. A chess master quickly identifies board positions and strategies from experience.
More generally, what gets memorized can take different forms
Facts: specific examples
Patterns: repeated structures
Rules: useful procedures or heuristics
Relations: associations and dependencies
So we can already see a more generic definition: memorization is retaining information from the past in a form that can be reused later.
Why AI can look creative
If AI is just memorizing, how can it produce outputs that seem new? How can it answer questions it has never seen or create images that never existed?
Because new outputs can be built from existing pieces.
Even when a request is new, the model has usually seen many related components: similar questions, nearby facts, common formats, and typical ways people solve problems.
The exact input and output may be new. But the process is still based on recombining stored patterns. Novelty doesn't always require deep understanding. Often, it only requires enough material to assemble a useful response in a new context.
Recombining patterns at scale is a big part of what makes modern AI powerful.
The real question
So the most interesting question is not "Can AI think like a human?"
The more useful question is: Why does memorization work so well for AI?
Why can systems trained on past data perform well on future tasks? Why are language, faces, preferences, markets, and decisions predictable enough for this approach to succeed?
This question will be answered in the next article: Why memorization works.
