Research Is Not a Substitute for Judgement
On the proper relationship between data and instinct in the making of great advertising.
In the autumn of 2004, I was summoned to a meeting at the London offices of a major multinational whose name, out of professional courtesy, I will not disclose. The meeting concerned a campaign that my team had been developing for the better part of four months. We had written the copy. We had art-directed the layouts. We had produced three television scripts. The work was, I believed then and believe now, among the strongest my agency had produced that year. It was also, the client informed us with evident satisfaction, about to be subjected to quantitative pre-testing. A research company would recruit four hundred consumers, show them our work in a controlled environment, and measure their responses across eleven proprietary metrics. The results would determine whether the campaign ran. Four months of thinking, crafting, and arguing — all of it reducible to a battery of scores on a spreadsheet.
The campaign scored well on ten of the eleven metrics. It failed on one: "purchase intent." Not by much — it fell two percentage points below the category benchmark. The client killed the work. Four months. Thousands of hours. Hundreds of thousands of pounds in fees. Destroyed by two percentage points on a single measure, in a methodology whose predictive validity no one in the room could have defended under cross-examination. I did not argue. I had learned, by that stage in my career, that arguing against research in a research-driven organisation is like arguing against weather. You may be right, but you will get wet regardless.
I tell this story not to suggest that research is useless. It is not. I tell it to illustrate a phenomenon that I have observed throughout my career with increasing alarm: the substitution of research for judgement. The two are not the same thing. Research is a tool — a powerful tool, when properly used, for understanding markets, identifying opportunities, and reducing certain kinds of risk. Judgement is the capacity to make decisions in conditions of uncertainty, drawing on experience, knowledge, intuition, and taste. (I explore the AI dimension of this problem in AI, Research, and the Death of Intuition.) Research informs judgement. It does not replace it. And yet, in agency after agency, client after client, the drift is unmistakably in one direction: towards a world in which no creative decision is made without data to justify it, and no piece of work is approved without a score to validate it.
The Research Paradox
There is a paradox at the heart of advertising research. (For my thoughts on how AI is reshaping this tension, see AI, Research, and the Death of Intuition.) There is a paradox that is rarely acknowledged by those who commission it. The purpose of research is to reduce risk. The purpose of great advertising is to take risks. These two objectives are fundamentally in tension. Research, by its nature, rewards the familiar. It rewards the expected. It rewards the safe. When you show a consumer a piece of advertising in a research environment and ask them whether they like it, they will reliably prefer the work that conforms to their existing expectations of what advertising should look like. The work that surprises them — that challenges them, that makes them uncomfortable, that forces them to revise their assumptions — will score poorly, because discomfort registers as a negative response in the blunt instruments of quantitative testing.
This is not a theoretical objection. It is an empirical observation, confirmed by decades of evidence. Some of the most successful campaigns in advertising history tested poorly. The Volkswagen "Think Small" campaign, which I have written about elsewhere, was reportedly received with scepticism in early qualitative research. Consumers found it confusing. They were accustomed to automobile advertisements that showed large, gleaming cars against aspirational backdrops. A small car, photographed small, against a white background, with a headline that seemed to be apologising for itself? It did not compute. Had the research been the final arbiter, the campaign would never have run. That it ran at all was due to the judgement of Bill Bernbach, who trusted his understanding of the cultural moment over the stated preferences of a roomful of respondents.
I do not wish to romanticise Bernbach's approach. He was not anti-research. He simply understood its limitations, which is a very different thing. He knew that research could tell you what people had liked in the past but could not tell you what they would respond to in the future, because genuinely original work creates its own context. The Volkswagen campaign did not succeed because it matched existing consumer preferences. It succeeded because it changed them. And no research methodology in the world can predict a change in preference, because the change is created by the very work the research is meant to evaluate.
What Research Can and Cannot Do
Let me be precise about what I believe research is good for, because I do not want to be misunderstood. I am not an enemy of data. I am an enemy of the misuse of data, which is a far more common problem.
Research is excellent at identifying problems. If your brand awareness is declining, research will tell you. If your target audience does not understand your proposition, research will tell you. If there is a gap in the market that no competitor is addressing, research will tell you. These are valuable insights, and any creative team that ignores them is being arrogant rather than brave.
Research is also useful — genuinely useful — at evaluating executional details. Should the pack be blue or green? Research can help. Should the television commercial be thirty seconds or sixty? Research can provide guidance. Should the headline emphasise price or quality? Research can offer evidence. These are decisions where the variables are limited, the context is stable, and the stakes are manageable. For decisions of this kind, research is a friend.
Where research fails — where it consistently, reliably, predictably fails — is in evaluating the quality of creative ideas. Not because the ideas are beyond evaluation, but because creative quality operates on dimensions that research instruments are not designed to measure. A great advertisement does not merely communicate a message. It communicates a message in a way that is memorable, distinctive, emotionally resonant, and culturally appropriate. These qualities are not additive. They are multiplicative. They interact in complex ways that cannot be captured by asking a respondent to rate their agreement with the statement "This advertisement is memorable" on a five-point scale.
I once sat through a research debrief in which a moderator reported that respondents found our television script "confusing." The script was for a luxury car brand. It opened with thirty seconds of silence — no dialogue, no voiceover, just the sound of the car on an empty road. The silence was the idea. It was designed to be arresting, unusual, different from everything else in the category. In the research environment, divorced from the context of an actual commercial break, divorced from the contrast with the noisy, cluttered advertisements that would surround it, the silence registered as confusion. In the real world, it would have registered as distinction. The research could not see the difference, because the research could not simulate the world.
The Judgement Question
If research cannot reliably evaluate creative quality, then what can? The answer, I am afraid, is judgement — and judgement is a concept that makes modern organisations profoundly uncomfortable. Judgement is individual. It is subjective. It cannot be audited or benchmarked or replicated. It belongs to people, not to processes. And in an era that worships processes, that trusts systems over individuals, that prefers the measurable to the meaningful, judgement feels dangerously old-fashioned.
But judgement is what built the advertising industry. Every great campaign in history was approved by someone who exercised judgement — who looked at the work, considered the evidence, weighed the risks, and said "yes" or "no" based on a combination of knowledge, experience, and instinct that no algorithm could replicate. Bill Bernbach had judgement. David Ogilvy had judgement. Mary Wells Lawrence had judgement. These were not people who ignored data. They were people who used data as one input among many, and who were willing to override the data when their understanding of the situation told them the data was wrong.
The decline of judgement in our industry is not, I think, a result of incompetence. It is a result of incentives. In a modern organisation, the person who approves a campaign that fails on the basis of their personal judgement is held responsible. The person who approves a campaign that fails on the basis of research is not. Research provides cover. It distributes blame. It allows the decision-maker to say "the data supported this decision" and to escape with their career intact. This is perfectly rational behaviour at the individual level. At the collective level, it is catastrophic, because it systematically biases the industry towards safe, mediocre, research-friendly work and against the bold, original, risky work that actually builds brands.
The Education of Instinct
I want to address an objection that I know will be raised: that judgement is unreliable. That it varies from person to person. That it is subject to bias, mood, and personal taste. All of this is true. But the conclusion that many people draw from these observations — that judgement should therefore be replaced by research — does not follow. The correct conclusion is that judgement should be educated.
The judgement of a creative director who has spent thirty years studying advertising, writing advertising, and observing the effects of advertising is not the same as the judgement of a junior account executive or a brand manager six months into the role. It is not infallible, but it is informed. It is calibrated by decades of experience — by seeing what works and what fails, by understanding why certain campaigns endure and others vanish, by absorbing the accumulated wisdom of the profession. This kind of educated judgement is not a guess. It is an informed assessment, and in many situations it is more reliable than the most sophisticated research, because it can account for nuances that research instruments cannot capture.
The great mistake of the modern advertising industry is the failure to cultivate this kind of judgement. We do not invest in developing the instincts of our senior people. We do not give them the time, the exposure, or the authority to develop a refined sense of what works. Instead, we ask them to defer to data — to set aside their accumulated knowledge in favour of a spreadsheet produced by a research company whose methodology was designed for a different purpose. This is not rigour. It is abdication.
I am reminded of a conversation I had many years ago with a wine merchant in Burgundy. I asked him how he decided which wines to stock. He said he tasted them. I asked whether he used any formal scoring system. He looked at me as though I had asked whether he used a thermometer to determine if he was in love. "I have been tasting wine for forty years," he said. "My palate is my instrument. Why would I replace it with a number?" He was not being arrogant. He was stating a fact. His palate was the product of four decades of deliberate practice. It was more sensitive, more discriminating, and more reliable than any scoring system, because it could detect qualities that no scoring system was designed to measure. The same principle applies to the judgement of a seasoned creative director — or it would, if we allowed it to.
A Proper Relationship
I do not advocate the abolition of research. I advocate a reformation of the relationship between research and creative judgement. Research should be a servant, not a master. It should inform decisions, not make them. It should be consulted before the creative process begins — to understand the market, the audience, the competitive landscape — and it should be used after the work is in market, to evaluate its actual effects. What it should not be used for is the intermediate stage: the moment when someone must look at a piece of creative work and decide whether it is good enough to run. That decision requires judgement, and judgement requires a human being who has earned the right to exercise it.
Ogilvy, who was himself a great researcher before he became a great copywriter, understood this distinction perfectly. He used research voraciously. He tested headlines, layouts, offers, and media placements with a rigour that few of his contemporaries could match. But he never outsourced the central creative decision to a research score. He trusted his own judgement, informed by data but not enslaved to it. He was, in this as in many things, a model that our industry has inexplicably abandoned.
The proper relationship between research and judgement is the relationship between a map and a navigator. The map tells you the terrain. The navigator decides the route. A navigator who ignores the map is reckless. A map without a navigator goes nowhere. The tragedy of modern advertising is that we have dispensed with the navigator and are attempting to let the map drive the car.
Research is not a substitute for judgement. It is the raw material from which judgement operates — and without someone willing to exercise that judgement, all the data in the world is merely noise.