Making Websites Win: the conversion rate optimization process

What does the book argue?

That the page is not the problem. The page is the symptom.

Blanks and Jesson call their method Scientific Web Design, and the shape of it is medical: diagnose before you prescribe. They structure the work as DiPS, for diagnose, problem, solution, and they spend most of the book on diagnosis, which is the part almost everybody skips. The chapters on research methods are the ones worth paying for: how to survey visitors who are leaving, how to interview people who bought and people who nearly bought, how to watch a session recording without talking yourself into a conclusion, and how to spot what they call Conversion Killers, the things on a page that stop a sale for reasons the owner cannot see any more.

Their best known case is Moz, then SEOmoz. They replaced a short promotional page with one roughly six times longer, built out of the founder's own in-person sales pitch, with a curiosity headline, a tier comparison, example expert answers, peer testimonials and a $1 trial. The first split test, over more than 5,000 visitors, lifted PRO membership sales 52%, the page reached roughly a 170% improvement over four months, and the company credits about $1 million in additional annual revenue to the work (Conversion Rate Experts case study). Note what happened there: the page got longer, which is the opposite of what most redesigns do.

What is the conversion rate optimization process, in practice?

This is the version I run. The order is the point.

Step 1: one page, one number, one week. Pick the single page with the most money flowing through it and name the one number you are trying to move, as a percentage with a denominator. "Visitors who start the form" beats "conversions". If you cannot say the number out loud, you are not ready to change the page.

Step 2: gather five inputs, in parallel, over five days. The analytics funnel for that page, broken by device and by traffic source. An on-page exit survey. Five customer conversations. Ten session recordings, watched in full. And the list of ads, search queries and referring pages that actually bring the traffic.

The survey and interview questions matter more than the method, so here are the ones that earn their keep:

• To people leaving the page: "What were you looking for that you didn't find?"

• To people who just bought or signed up: "What nearly stopped you today?"

• To new customers, in an interview: "What were you using before this, and what happened that made you go looking?"

• To people who cancelled: "What did you expect this to do that it didn't?"

The second question is the one that pays. People will tell you the exact objection they overcame, and it is almost never the one the page addresses.

Step 3: write the objections down in the customer's words. Not your paraphrase. The output of the research is a ranked list of objections, not a list of ideas. Rank them by how often they came up and how expensive they are to answer.

Step 4: map each objection to a place on the page. For each one, pick one of three moves: answer it, prove it, or remove the thing that caused it. Most teams only ever do the first, which is how pages get long without getting better.

Step 5: build one big change, not five small ones. This is where low traffic decides your method, and the book does not help you much here. More on that below.

Step 6: pick the test method from your traffic, before you build. If you can reach sample size, run an A/B test. If you cannot, run the change as a before and after over at least four weeks, holding the traffic mix constant, and write down in advance what result would make you keep it. Deciding this before you build stops the argument about whether the result counts.

What do you do when you do not have the traffic for A/B tests?

You make bigger changes, and you accept that the arithmetic is not negotiable.

Here is the arithmetic. To detect a change at 95% confidence and 80% power, the approximate visitors needed per variant is 16 multiplied by p multiplied by (1 minus p), divided by the square of the absolute difference you are trying to detect. Put real numbers in it:

• A page converting at 2%, trying to detect a 25% relative lift to 2.5%: about 12,500 visitors per variant, so 25,000 in all. At 1,000 visitors a week, that is 25 weeks.

• The same page, trying to detect a 50% relative lift to 3%: about 3,100 per variant, so 6,300 in all. At 1,000 visitors a week, about 6 weeks.

Doubling the size of the change you are willing to attempt cuts the test to a quarter of the length, because required traffic scales with roughly the inverse square of the effect you are chasing. That single fact should decide how you work. Five button tests at 1,000 visitors a week will take you two years and tell you nothing. One rebuilt page, tested against the old one, will tell you something before the quarter ends. Evan Miller's sample size calculator will do the sum for your own numbers in a minute.

There is a second reason to make fewer, bigger changes. Most tests lose. At Microsoft, one third of experiments proved effective, one third were neutral and one third were negative, and at Google and Bing only about 10% to 20% of experiments produced positive results (Kohavi and Thomke, Harvard Business Review, September-October 2017). Those are teams with world-class research behind every idea. If your hit rate is similar, and there is no reason it would be better, then the number of attempts you can afford is the real constraint, and each one had better be worth the wait.

What has dated?

The tools. The recommendations are from 2017, and most of the products named have changed, merged or closed. Read those chapters for what the tool is for, not for which one to buy.

The traffic assumption. Almost every example has enough volume to split test quickly. A seed-stage B2B company with 800 visitors a week cannot run the book's process as written, and the book does not tell you what to do instead.

The business types. The examples lean heavily on transactional ecommerce and high-volume lead generation. If you sell to a committee over four months, the "objection" model still holds, but the page is one surface in a long conversation rather than the place the decision happens.

What is missing entirely. Consent banners and the measurement they broke. Mobile-first behavior. App onboarding. And AI answer engines, which now send a smaller number of much better informed visitors, who arrive with the basic objections already answered and the specific ones still open. That last change makes the research step more useful than it was in 2017, not less.

Who should read it?

Anyone who owns a page with real money going to it and has never done the research step. Read part two, on diagnosis, properly, and skim the rest. Skip the book if you do not have a page yet, or if your traffic is still so small that the only honest research is talking to people, in which case have the conversations and come back to the book when there is something to measure.