Visual regression testing for TYPO3 upgrades with BackstopJS

Putting the website through its paces after a TYPO3 upgrade is one of the most time-consuming and stressful tasks in an integrator's day-to-day work. Clicking through every page manually, checking every content type, checking every view - this can quickly eat up hours or even days. And yet the queasy feeling creeps in: Have I really got everything?

Many integrators therefore create special test pages on which all content elements are bundled together - a pragmatic approach to checking the most important components at a glance. But even this does not cover all eventualities: different template contexts, nested structures, special pages with special cases. The test page shows that the text-image element works - but not whether it looks the same on every real page.

Over the past few months, I have developed a tool that alleviates precisely this pain: An automated workflow for visual regression testing with BackstopJS. What used to be manual and error-prone is now structured and reproducible.

Yes, I know: If you have a fully automated CI/CD pipeline with Playwright, Chromatic and Kubernetes, you don't need this. Anyone who has been using Percy or Applitools in the enterprise setup for years doesn't anyway. This tool is for the others - the freelancers and small agencies that don't have a DevOps department and still want to test professionally. Without three weeks of setup time.

The problem: Manual testing after upgrades is inefficient

Anyone who carries out TYPO3 upgrades knows the process: Composer update, database migration, clearing the cache - that's the technical side. But then comes the actual stress test: Does the website still work as before?

The classic method looks like this: You click through the website, open different pages, check different content elements, check different templates. For a small website with 20 pages, this is still manageable. With 200 or 500 pages, it becomes an ordeal.

And it's not just about obvious errors such as completely broken layouts. The nasty bugs are the subtle ones: A content element that has slipped two pixels to the right. A missing space between headline and text. An image that is suddenly no longer left-aligned but centered.

These visual deviations are only noticeable when you look closely - or when the customer calls and asks: "Why does this look different now?"

Visual regression testing: the idea behind it

Visual regression testing compares screenshots of a website before and after a change. The software takes images of the current state (reference) and the new state (test), superimposes them and marks all differences with pixel accuracy.

What the tool cannot do: Tell you whether the change is intentional or a bug. But it shows you every deviation - and you decide whether it is okay or needs to be fixed.

BackstopJS is such a tool for visual regression testing. It is based on Node.js, uses headless browsers (Chrome/Firefox) and automatically creates screenshots in different resolutions. The generated reports show you at a glance which pages have changed and what the differences look like.

Why I built my own tool around it

BackstopJS is powerful, but not immediately ready for larger TYPO3 websites. Manual configuration can get cumbersome, especially with dozens or hundreds of URLs.

That's why I've developed an automated workflow that solves three key problems:

1. automatic URL crawling

Instead of collecting URLs manually, a PHP script crawls the website and writes all URLs found to a CSV file. Alternatively, since version 1.1.0 you can also have an existing XML sitemap read out - this is much faster than a complete crawl, but of course only works if the website offers a sitemap. This saves time and ensures that no page is overlooked.

2. batch processing for large websites

Creating screenshots of 500 URLs at the same time would overload the system. The tool automatically divides the URLs into manageable batches of 40 pages and processes them one after the other.

3. structured workflow management

A command line tool guides you through the entire process: activate the next batch, create the screenshots, compare the results, mark the batch as complete. This allows you to keep track of large projects.

My typical workflow for TYPO3 upgrades

How I use the tool in practice:

Step 1: Create a local copy

I create a local copy of the project with DDEV, carry out the upgrade and make sure that everything is technically up and running. Composer updates, database migration, adapting the code - the usual steps.

Step 2: Crawl the website

I use the crawler script to fetch all the URLs of the live website. I have two options here:

Option 1: Perform a complete crawl

ddev exec php crawler.php --url example.com>

The script crawls the website, automatically filters non-HTML files (PDFs, images) and writes all pages found to a CSV file. The whole thing runs in real time - URLs are written immediately, no memory problems with large sites.

At the end I get a detailed error overview: which pages return 404, which 403, where there were timeouts. This may already give me some initial clues as to where I should take a closer look.

Option 2: Read XML sitemap (since version 1.1.0)

ddev exec php crawler.php --sitemap example.com/sitemap.xml

If the website offers an XML sitemap, this is significantly faster. The script reads the sitemap and extracts all URLs directly - without going through every single link on every page. This also works with nested sitemaps: Sitemap groups are automatically recognized and run through recursively.

Step 3: Generate test scenarios

I use the collected URLs to automatically create BackstopJS scenarios:

ddev exec php create-backstop-scenarios.php \\
  --test=http://example.ddev.site \\
  --reference=https://example.com

The script splits the URLs into batches and creates a separate scenario file in the scenarios/pending/ folder for each batch.

Step 4: Test batch by batch

Now comes the interactive part. I activate the first batch:

ddev exec php manage-scenarios.php next

Then I create the reference screenshots from the live site and the test screenshots from my local instance:

backstop reference --config ./backstop.js
backstop test --config ./backstop.js

BackstopJS automatically opens an HTML report in the browser. There I can see for each URL:

• A before image (live site)
• An after image (local instance)
• A diff view with highlighted differences

I go through the results and decide for each deviation: Is this a bug or an intentional change? If everything is okay, I mark the batch as done:

ddev exec php manage-scenarios.php done

Then I activate the next batch and repeat the process. With 200 URLs, this means five runs with 40 URLs each - structured and without losing the overview.

What the tool actually does for me

Time saving

Of course, for large websites, the entire process can take one to two hours. But compared to manually clicking through 200+ pages, that's a joke. And above all: the software doesn't miss anything. No more "I didn't see that".

Security

I have a good feeling when I upload the tested website to the production environment. The screenshots show in black and white: it looks the same as before. Or they show where there are problems - then I can fix them before the go-live.

Documentation

The screenshots and reports are also valuable later on. If a customer says "That looked different before" months after the upgrade, I can dig out the old reference screenshots and show them: No, it looked exactly the same.

Where the tool has its limits

I want to be honest: The tool is not perfect. It has weaknesses.

Dependence on DDEV

The workflow is currently based on DDEV. This means that you need either DDEV or a native PHP on your system. If you work with other setups, you will need to make adjustments.

Manual interpretation required

The tool shows you every deviation - even harmless ones. A cookie banner that behaves differently the first time it is called up. A dynamic element that loads random content. A timestamp that is naturally different. You have to decide for yourself what is relevant and what is not.

Configuration effort for complex sites

If a website has many dynamic elements (login areas, forms, JavaScript-heavy components), you need to adjust the BackstopJS configuration. Hide certain selectors, increase delays, define interactions. That can be tricky.

Nevertheless: For my purposes, the tool works well enough that I use it for every major TYPO3 upgrade.

Who is the tool suitable for?

If you manage TYPO3 websites and regularly perform upgrades, the tool is worth a look. Whether as a freelancer with smaller projects or in an agency with larger clients - structured testing always helps.

It is particularly useful if you:

• manage websites with many pages and different templates
• are upgrading across several TYPO3 versions (e.g. from 9 to 12)
• have clients who pay very close attention to visual consistency
• want to professionalize your testing processes

The technical requirements are straightforward: you should be able to work with DDEV, Docker and the command line. BackstopJS can be installed with npm install -g backstopjs. If you are already working with this stack, the learning curve is flat.

Test the tool and give feedback

The repository is publicly available on GitHub: https://github.com/wwagner-net/create-backstop-scenarios

There you will find:

• A detailed README with all commands
• Step-by-step instructions for getting started
• Explanations of all scripts and options
• Examples for different use cases

The tool is under MIT license, you can use, adapt and extend it freely.

I would be happy if you try it out and share your experiences with me:

• Does the workflow work for your projects?
• Where does it get stuck?
• What are you missing?
• What improvements would you like to see?

You are welcome to send me feedback via an issue on GitHub. I am continuing to develop the tool and welcome suggestions from the field.

Conclusion: Not a miracle cure, but a useful helper

Visual regression testing does not replace manual testing - but it does make it much more efficient. Instead of blindly clicking through hundreds of pages, you get a structured overview of all visual changes.

The tool I developed makes BackstopJS manageable for larger TYPO3 projects. It automates the crawling, organizes the processing in batches and guides you through the workflow in a structured way.

Is it perfect? No. Does it have weaknesses? Yes. But it helps me sleep better every time I upgrade TYPO3 - because I know I haven't overlooked anything important.

If you test the tool, let me know how it goes for you. I look forward to hearing about your experiences.