andydataguy
DATA & DECISIONS . Agency Operations . 2021

Programmatic Split-Testing Engine

100+ Weekly Automated PPC Tests

Andy Houston author chipBY ANDY HOUSTON
Test-creation factory conveyor stamping experiments past a QA gateT1T2T3T4T5T6T7T8T9T10T11T12T13T14QA GATETEMPLATES IN100+ TESTS / WEEKDEFAULT BEHAVIOR. NOT HEROIC EFFORT.
100+
weekly tests
Compounding
CTR / CPA / LTV
Default
behavior
Central
log

Context

Performance marketing environment where large budgets demanded constant experimentation across keywords, creatives, and landing pages.

Problem

Manual testing could not keep up with the volume of hypotheses. Important tests died in spreadsheets. Results were hard to track, and learnings were rarely reused.

Approach

Designed a programmatic split-testing engine: structured naming conventions, automated campaign creation, and templated experiment setups that made it easy to spin up, monitor, and retire tests every week. Results flowed into a central log of "wins," "losses," and "context."

Weekly-test-volume bar chart climbing past 100/week with win/loss/context labels.

TEST VOLUME . WEEKLY CADENCE100 / WEEK FLOOR22W1CTX38W2W54W3W71W4L88W5W102W6W118W7L124W8W132W9W
RECEIPT FILM . THE ARC BEFORE TO AFTER
SCROLL TO SCRUB

The before-to-after, silent-legible: where it started, what the data showed, the decision that turned it, where it landed. Past performance guarantees nothing. The diagnosis method is the product.

RECEIPT · PROGRAMMATIC-SPLIT-TESTINGBEFORE / AFTERWHERE IT STARTEDManualthe testing cadenceImportant tests died in spreadsheets, and learnings went unused.WHAT THE DATA SHOWEDEVERY TEST INTO ONE SHARED LOGkeywordcreativelanding pageTests died in spreadsheets, so the learnings were never reused.THE TURNWe templated the experiment so spinning one up cost nothing.WHERE IT LANDED100+tests a week, every result loggedTHE METHOD IS THE PRODUCTPast performance guarantees nothing. The diagnosis method is the product.

Stack

  • Google Ads
  • Facebook Ads
  • Scripts and automations
  • Shared experiment database

Result

100+
weekly tests

Supported 100+ weekly tests without the team drowning. Over time, the engine identified compounding gains in CTR, CPA, and LTV that would have been impossible to find by hand.

100+ · WEEKLY TESTS

Compounding-gains chart. CTR, CPA, LTV trends across thousands of tests.

WEEKLY TESTSCTRCPA INVLTV100+WEEKLY TESTS

Impact

Turned testing from a heroic extra effort into a default behavior of the account. The system made it harder NOT to test.

Lessons

At scale, your advantage is the machine that runs thousands of tests and remembers what worked.

Why this matters to you

For any team spending enough on ads that "winging it" with a few monthly tests is leaving serious money on the table.

Sounds like you? Get a quote