- Steven Cravotta
- Posts
- Don't AB test unless you have this
Don't AB test unless you have this
Hey there,
You ran an AB test on your paywall. One version won, and you're already rebuilding around it.
Don’t do this.
If you tested on just a handful of installs, the result means nothing.
A handful of installs is not a real signal.
You're reading noise and calling it a data-backed decision.
This is the trap at the early stage.
You launch, downloads trickle in, conversions look low, and the instinct kicks in to start fixing everything.
So you spin up five paywall variants, rebuild the onboarding, swap the pricing.
None of it tells you anything, so you burn weeks rebuilding around a result that was never real.
Let's use the coin flip example.
Flip it 10 times. You might get 8 heads and decide heads is better.
Flip it 1,000 times and it evens out.
Your funnel works the exact same way.
When I scaled PuffCount to $44k a month, I didn't split-test until hundreds of people were hitting my paywall every week.
Once the volume was there, the tests finally told me something real.
Instead of split testing too early, here's what I do.
Get more people in first. Pour everything into content and ads until real numbers are flowing through your funnel.
Only then is it time to split-test.
With volume underneath it, if you see one paywall convert better, you know it’s a real signal.
Remember, the goal is a paywall that prints.
And it can't print until enough people run through it.
Peace,
Steve