Data
The Un-SaaS Ch. 13 — The Experiment Log
The operator who runs one experiment per week for a year has 52 data points no competitor can see. The log is the moat.
September 18, 2026 · 11 min read
Chapter 13 of 26
This is a chapter from The Un-SaaS: A Toll Stack Engineer's Handbook. Each Friday, we're publishing a chapter as a bonus for our readers.
In 1945, Vannevar Bush published an essay called “As We May Think.” Bush was the engineer who ran the US Office of Scientific Research during World War II — not a theorist, a builder. He proposed a device called the memex: a desk-sized machine that would store a researcher’s notes, references, and observations. The key feature was that entries were linked by association rather than alphabetical order.
The memex was never built. But the idea behind it was simple and powerful: the value of research isn’t in individual experiments. It’s in the accumulated web of connections between them.
An experiment that shows “Subject line A beats Subject line B by 12%” is useful. An experiment log that surfaces the connections between experiments is something else entirely — that’s a moat. “Subject lines with numbers outperform subject lines with questions by 15%, except on Tuesdays when the audience opens emails at 6 AM instead of 9 AM, in which case question-format subjects outperform number-format subjects by 8%.”
Nobody can copy your experiment log. They can copy your landing page. They can read your email subject lines. They can reverse-engineer your offer structure. But the web of connections between eight hundred logged experiments is something else. The pattern intelligence that tells you not just what works, but what works for whom, when, and why — that can only be built by running the experiments yourself, over time.
The experiment log is the thing that makes month eight pay ten times what month one paid. And it’s the thing that makes your toll position genuinely irreplaceable.
The Weekly Metrics Ritual
Every operator needs a weekly practice. Not a daily dashboard addiction. Not a monthly retrospective. A weekly ritual that takes nine minutes and tells you three things:
Question 1: Is the machine working?
Four numbers:
- Capture rate: What percentage of landing page visitors entered their email? (Target: 25-35%)
- Sequence completion rate: What percentage of new subscribers received all emails in the welcome sequence without unsubscribing? (Target: 70-85%)
- Click-through rate: What percentage of email recipients clicked on a product link? (Target: 3-8%)
- Conversion rate: What percentage of clicks resulted in a purchase? (Target: 2-5%)
If any of these four numbers drops below its target range, something is broken. Diagnose it before running any new experiments.
Question 2: Is it growing?
Four numbers:
- New subscribers this week: Absolute count. Trending up, flat, or down?
- Active list size: Total subscribers minus unsubscribes and bounces.
- Revenue per subscriber per month (RPS): Total monthly revenue divided by active list size. (Target: $1.50-$2.50 for a single position, $2.00-$3.50 at five-plus positions)
- Weekly revenue: Total revenue across all layers for this position.
These four numbers tell you whether the position is compounding or stagnating.
Question 3: Will anything break?
Four numbers:
- Unsubscribe rate: Weekly unsubscribes as a percentage of sends. (Warning threshold: above 0.5% per send)
- Bounce rate: Hard bounces as a percentage of sends. (Warning threshold: above 2%)
- Partner traffic trend: Is the partner’s content output consistent? Declining output = declining captures.
- Experiment velocity: How many tests completed this week? (Target: 1-2 per week per position)
Twelve numbers. Three questions. Nine minutes. If all twelve are in range, close the dashboard and go do something else. If one is out of range, it’s tonight’s ninety-minute session.
The Monday Morning Check
Here’s how the ritual works in practice. Every Monday morning — or Friday afternoon, if you prefer closing the week instead of opening it — you sit down with three tabs open: your email platform, your redirect dashboard, and your analytics.
Fifteen minutes. Not thirty. Not an hour of “deep diving into the data.” Fifteen minutes of checking twelve numbers against their thresholds.
The three groups map to an escalation protocol. When a number crosses its threshold for one week, you note it and watch. Might be noise — a holiday week, a one-off technical glitch. When it crosses for two consecutive weeks, you investigate. Pull detailed data. Identify the specific email, product, or segment causing the deviation. When it crosses for three consecutive weeks, you act. The problem is structural, not momentary.
A deliverability drop from 96% to 88% on a 4,000-person list means 320 people aren’t seeing your emails. Three weeks of that produces 180 cold subscribers and thousands in lost revenue — invisible unless you checked the specific number. Reactive measurement catches that problem after it’s been compounding for a month. The Monday morning check catches it in week one.
Most problems resolve at yellow. They were noise. The ritual confirms health forty-eight weeks a year and catches the four weeks of real problems before they compound into month-long revenue losses.
The Experiment Framework
An experiment in the toll position context is a structured test with five components:
Hypothesis: “Changing [X] will improve [metric] by [estimated percentage] because [reasoning].”
Variant: The specific change being tested. A new headline. A different subject line. A reordered email sequence. A moved CTA button. A changed price anchor.
Sample size: The number of visitors or subscribers needed to reach statistical significance. For landing page tests, typically 200-500 visitors per variant. For email tests, typically 500-1,000 recipients per variant.
Duration: How long the test runs. Typically one to two weeks for landing page tests, one to two sends for email tests.
Result: Winner, loser, or inconclusive. Plus the specific metric change: “Variant B improved capture rate from 27% to 31%, p < 0.05, over 2,400 visitors per variant.”
Every experiment gets a row in the log. Winners, losers, and inconclusive results all matter — because knowing what doesn’t work is as valuable as knowing what does.
What to Test (In Order)
New operators often ask: “What should I test first?” The answer is a priority stack based on leverage — changes that affect the most revenue per hour of work.
Tier 1: Test first (highest leverage).
- Landing page headline. The headline determines whether the visitor stays or bounces. A 10% improvement in bounce rate flows through to every downstream metric.
- Email subject lines on the welcome sequence. Open rate determines whether anyone reads your content. A subject line that lifts open rate by 8% means 8% more people see your product recommendation.
- The CTA on the email capture form. “Get the full analysis” vs. “Subscribe to updates” can produce 15-25% differences in capture rate.
Tier 2: Test second (medium leverage).
- Pre-sell slide content and order. Which slide hooks them? Which slide makes them leave?
- Email send timing. Morning vs. evening. Weekday vs. weekend. The optimal time varies by niche and audience.
- Product recommendation framing. “I recommend this because…” vs. “Here’s what I use…” vs. “The data suggests…” — different audiences respond to different authority frames.
Tier 3: Test third (lower leverage, still worth doing).
- Landing page visual layout. Image vs. no image. Long page vs. short page. Video embed vs. text-only.
- Email sequence length. Five emails vs. seven. Fourteen days vs. twenty-one.
- Secondary offer placement. Which email in the sequence is the right spot for the second product recommendation?
One experiment per tier, completed before moving to the next. Most operators spend months in Tier 1 because the leverage is so high. That’s correct.
The Compounding Effect
Here’s where the experiment log becomes a moat.
Each individual experiment produces a small improvement — maybe 2-5% on the metric being tested. That sounds trivial. It’s not.
A 5% improvement in capture rate means 5% more subscribers per month. Those subscribers flow through the welcome sequence, earning revenue. And they flow through it every month going forward, not just the month you ran the test.
So a single experiment that improves capture rate by 5% in month three produces additional revenue in months 4, 5, 6, 7, 8, 9, 10, 11, and 12. One afternoon of work in month three generates nine months of incremental revenue.
Now stack that. Two experiments per month, each producing a 3% improvement on a different metric. After twelve months, you’ve run twenty-four experiments.
The compound math is precise: 1.03 raised to the 24th power equals 2.03. Two experiments per month at 3% each doubles your baseline in one year. A landing page at 18% capture becomes 36%. An email sequence at $1.20 RPS generates $2.44. A bridge producing $1,800 a month now produces $3,660 from the same traffic, same niche, same partner.
Not every experiment produces a winner. About 40-60% of well-designed experiments yield a measurable improvement. The rest tell you what doesn’t work — equally valuable intelligence that narrows the optimization space for future tests. The real ledger: twenty-four experiments, twelve winners at 3-5% each, twelve lessons that prevent mistakes you would have otherwise repeated.
The compound effect of those twenty-four small improvements — each building on the last — produces a total lift that’s dramatically larger than any individual test.
This is why month eight pays ten times what month one paid. It’s not because the traffic increased. It’s not because the product changed. It’s because the experiment log has nine months of intelligence encoded into the infrastructure, and every new subscriber benefits from every past experiment simultaneously.
A competitor who copies your current landing page is copying a snapshot — the result of nine months of work. They’d need to run their own nine months of experiments to catch up. By then, you’re eighteen months ahead.
That’s the moat. Not code. Not design. Data.
A/B Testing Discipline
One more thing that separates serious operators from hobbyists: testing discipline.
Run one test at a time per position. Multiple simultaneous tests confuse the results — you can’t tell which change caused the improvement.
Wait for statistical significance. Don’t call a winner at fifty visitors. The minimum is usually two hundred per variant, and for email tests, five hundred per variant. Premature winner declarations produce false positives that can actually decrease revenue.
Log every result, including losers. The losing experiments are as valuable as the winners — they tell you what the audience doesn’t respond to, which narrows the search space for future tests.
Trust the data over your instincts. You will be wrong about which variant wins roughly 40% of the time. That’s fine. The experiment log doesn’t care about your intuition. It cares about the numbers.
Never call a test before running it for a full seven days. Day-of-week effects are real. Creator audiences don’t behave the same on Tuesday as on Saturday. A test that “wins” Monday through Thursday might lose when weekend traffic arrives. The minimum sample sizes: 200-500 visitors per variant for landing page tests, 500-1,000 recipients for email tests, 50+ conversions for offer tests. Below those thresholds, you’re reading tea leaves, not data.
Let agents handle the mechanics. AI agents can draft variant copy, monitor test results, promote winners, and revert losers. Your job is to choose what to test, review the results, and decide what to test next. The strategic judgment stays with you. The execution goes to the agents.
The Experiment Schema
Every experiment in the log follows the same five-field schema. Not because structure is fun, but because the experiment you ran in month three needs to be findable and interpretable in month twelve. That means findable by you, by an agent, or by a buyer who’s evaluating your position as an acquisition target.
1. Hypothesis. Written before touching any tool. “Curiosity-framed subject lines will outperform clarity-framed for broadcast emails to month two-to-four subscribers.” If you can’t write a specific hypothesis, you don’t know enough to run the test.
2. Single variable. Only one thing changes between variants. If you change the headline and the CTA, you can’t isolate which change produced the result. An operator testing button colors before fixing a broken headline is optimizing the checkout flow of a store nobody walks into.
3. Measurement period. Minimum seven days. Maximum depends on traffic — landing page tests typically need one to two weeks, email tests need one to two sends across significant list segments.
4. Sample size. The specific number required to reach statistical significance. Not a guess — a calculation based on the expected effect size. Large effects (10+ percentage points): 300-500 visitors per variant. Medium effects (5-10 points): 700-1,000. Small effects (2-5 points): 1,500 or more.
5. Logged outcome. Winner, loser, or inconclusive. The specific metric change with numbers attached: “Variant B improved capture rate from 27% to 31% over 412 visitors per variant.” Winners, losers, and inconclusive results all get logged. Knowing what doesn’t work narrows the search space for every future test.
The 200-Experiment Moat
Here’s where the experiment log stops being a productivity habit and starts being a genuine competitive asset.
An operator with two hundred logged experiments across twelve months doesn’t just have a better landing page. They have a map of their audience’s decision architecture. Which emotional triggers move them. Which proof formats they trust. What time they open email. Whether curiosity or clarity works better in subject lines. Whether short or long pre-sells convert higher for this specific niche-partner combination.
That intelligence transfers across positions. When you open a new position in an adjacent niche, 60-70% of what you learned applies. Your second bridge reaches optimization in half the time. Your third in a third. The experiment log is the toll position’s most valuable infrastructure — more valuable than the landing page, the email sequence, or the redirect layer — because all of those can be rebuilt in a weekend. The intelligence behind them can’t.
A buyer evaluating your portfolio sees the same thing. Standard acquisition multiples for a toll position run 24-36x monthly net revenue. A $5,000/month position sells for $120,000 to $180,000. Add a documented intelligence database with cross-purchase patterns, audience quality scores, and eighteen months of optimization data, and the multiplier carries a 15-30% premium. Same position, same revenue — but the data moat adds $18,000 to $54,000 in acquisition value. Extra value from documenting what you already learned.
The experiment log is not optional. It’s the core of the entire model. Without it, a toll position is a static landing page that slowly decays. With it, a toll position is a self-improving system that gets better every week.
Next Friday: Ten days from first contact to live toll position. The sprint protocol.
Data
Want to build your intelligence moat?
This article covered the data dimension. The newsletter delivers measurement frameworks and signal patterns every week.
Unsubscribe in one click. Useful operator material only. Promotions, sponsors, and partner links will be disclosed.