Why We Built Our Own Testing Platform
We kept losing time to procurement.
The engineering case for visual regression and Lighthouse gating on a storefront takes an afternoon to make. The off-the-shelf tools are good and we would recommend several of them. Then the request goes into a vendor queue at a large client and sits there, because adding a SaaS product means a security questionnaire, a legal review, a question about where screenshots of unreleased pages are stored and who can see them, a new contract, a new billing relationship, and a new set of accounts to offboard when people leave. None of that is unreasonable. It is just slow.
Meanwhile the client already has an AWS account, an approved AWS relationship, and a team who operates it every day. Screenshots of a pre-release page are much easier to keep inside that boundary than to argue about moving out of it.
So we built the tests we wanted to run as a service that deploys into the client's own account.
What it is
A serverless platform for catching unexpected visual changes and performance regressions before they ship.
It compares screenshots captured by CI on deploy previews against the branch's approved baseline. It also accepts Lighthouse CI results, checks them against performance budgets, and tracks trends for each page.
In the dashboard, reviewers can inspect and approve or reject visual changes, set performance budgets, choose which pages CI checks, and see who did what.
Visual and performance checks each post a status on the pull request, linking to the results in the dashboard. Teams can make either check required or keep it advisory while they tune it.
How it finds issues
For the visual check, each screenshot CI uploads is laid over the approved one for the same page and compared pixel by pixel. We kept the comparison deliberately simple. Text renders slightly differently from one machine to the next, so the edges of letters are ignored. When something has changed, the diff notes which stripes of the page it fell in, so a reviewer can jump straight to it on a long page instead of scrolling. And if the page has come out a different height, which usually means a section has disappeared, that is reported as a change in its own right, not as a comparison that could not run.
Each page ends up as unchanged, changed, new or removed. A build only passes when every page is unchanged. Anything else, a new page included, puts it into changes_pending, the PR check goes red, and it stays red until someone has looked. Approve, and the new screenshots become the baseline. Reject, and the old ones stay.
This one is our own site. A pull request came back with the Notes and Shop links gone from the header, on every page, and nothing in the pull request had been meant to touch the header. Onion-skinning the baseline over the current capture shows it straight away, and side by side confirms nothing else moved.
For performance, the uploads come from the standard Lighthouse CI client, so there is nothing bespoke to install in a client's pipeline. The budgets are kept on the server: a minimum score per category, and a ceiling on the handful of metrics that matter, applied to the median of each page's runs. A team can get the uploads flowing first and switch the gate on once they have real numbers to set it from.
Those budgets need headroom. Lighthouse in CI is a lab number from one machine, as the docs say, and a budget pinned to the typical result will flake. The first set we seeded for a client was deliberately looser than the thresholds it replaced, and stays advisory until the medians have had time to settle.
Trends are kept per page and per device, not per deploy preview. Every commit gets a preview on a new hostname, and if the hostname were the key, every build would start a fresh line. Keeping /collections/shop-all as one line across builds means a slow regression shows up as a drift over weeks instead of hiding in the noise.
Never leave a PR stuck
A check that sometimes leaves a PR waiting forever is worse than no check at all. So every build ends in a status, including the ones whose worker died halfway through: those are marked as failed on the PR and asked to run again, rather than sitting on pending until someone goes looking.
If the platform itself is down or slow, CI notices, falls back to how it worked before, and carries on. The platform can fail. The client's pipeline should not.
It runs in the client's AWS account
This is the part that solved the original problem. The whole thing is an SST app and deploys with one command into whichever client account the operator is signed into.
Everything it creates is a managed AWS service the client's team already runs. Lambda for the API and the workers, S3 for screenshots and reports, DynamoDB for the records, CloudFront in front of the dashboard, Cognito for sign-in.
There are no servers to patch and nothing running between builds. The repo holds no account IDs, so the same code deploys into any client's account under their own credentials. Screenshots of unreleased work never leave that account, access follows the client's existing controls, and every reviewer action lands in an audit trail.
From a procurement point of view, nothing was bought. It is code running on infrastructure the client already pays for and has already approved.
What it cost us
We own the maintenance now, and the runbooks, and keeping up with the Lighthouse CI client. We pinned the scope hard to keep that bearable: visual checks, performance checks, one dashboard, and an infrastructure diff reviewed before every deploy. It is not a feature-for-feature replacement for a mature SaaS product.
For the clients it was built for, though, the realistic alternative was no tool for months. What moved the conversation from procurement to engineering was being able to say this:
That is the conversation we are good at.
If you want visual regression and performance gating without another vendor to procure, speak to us at work@allgold.com.