Visual regression testing using Jest, Chromeless and AWS Lambda

Testing is a crucial element of a great digital product, but it’s hard to test for visual changes in a website. In this post, we show you how we do it.

Visual changes in your website are hard to test. Even if the HTML of a certain component looks identical, a small change in CSS can have a huge impact on the visualisation of your website – which can lead to all sorts of problems.

Visual Regression testing tries to solve this problem by taking a screenshot of the UI and comparing that with a previous version. In short, if the difference exceeds a certain threshold, the test will fail.

A little background

A well-known tool for visual regression testing today is PhantomCSS. It uses PhantomJS, which is “a headless WebKit scriptable with a JavaScript API”. In other words, it’s a browser without graphical user interface that can be controlled programmatically.

Using this API, we can navigate to a web page and compare screenshots using PhantomCSS, allowing us to write visual regression tests like this:

phantomcss.screenshot('#the-dialog', 'a screenshot of my dialog');

For some time, this was certainly the way to go – and then Google released headless Chrome. As the name suggests, this is a way to run Chrome in a headless environment, allowing us to control it using the Chrome devtools protocol.

Soon after this announcement, one of the core maintainers of PhantomJS threw in the towel and stopped maintaining his library, saying: “I think people will switch to it, eventually. Chrome is faster and more stable than PhantomJS. And it doesn’t eat memory like crazy.” A fair assessment, and soon two great Node.js APIs on top of Headless Chrome showed up: Chromeless and Puppeteer.

Chromeless is released by Graphcool, while Puppeteer is released by Google, but they are both Node.js libraries that provide an API to control Chrome in headless or non-headless mode. At the moment, they both use the Chrome DevTools Protocol, but there are a number of differences. For instance, Chromeless gives you the ability to run Chrome in headless mode on AWS Lambda, while Puppeteer doesn’t. The advantage of this approach is that you can run your test locally (in parallel) and have them executed remotely on AWS Lambda. On the other hand, the API of Puppeteer is more extensive than Chromeless’s.

In the future, Chromeless will use Puppeteer under the hood and focus more on higher-level abstraction and remote proxy support (e.g. AWS Lambda).

So when it came to defining our approach to visual regression testing, we wanted to try Chromeless. Chromeless gives us AWS Lambdas out of the box, and will focus more on higher-level abstraction. Currently, we are using Jest at November Five, to test our React apps, so we wanted to maintain visual regression tests with the same library.

Let’s get started

First of all, we need to have a site to test. Because we don’t want to waste our time creating and setting up a website, we are going to use a static site framework that was recently released: React Static.

Start by installing and creating your static site using the following commands:

yarn global add react-static
react-static create

Then navigate to the created folder and check if everything is alright by running your development server yarn start.

It’s important to note that in the next sections we will use the async/await syntax. If you’re not using the react-static project as a starting point, make sure that your .babelrc configuration file enables the transform-async-to-generator plugin.

Jest & Chromeless

Now that we have our static site running, let’s write some tests! Install Jest and Chromeless using Yarn:

yarn add --dev jest chromeless

Next, we need to define a test script in the package.json which will run our Jest tests. A current limitation of Chromeless is that we can’t run our tests in parallel on our local machine. In order to run our tests sequentially, we need to pass the --runInBand option to Jest. We also provide a setup file with some default settings for our tests.

  "scripts": {
    "test": "jest --runInBand --setupFiles='<rootDir>/jest/setup.default.js'"

// jest/setup.default.js
global.config = {
  chromeless: {},
  baseUrl: 'http://localhost:3000',

Now we’re almost ready to write our tests. Because we want to keep our tests DRY, we create a utility file that initiates and terminates Chrome. We use the config defined in our previous setup file to define our Chromeless options. Right now, this is just an empty object.

Note that we also change the default Jest timeout from 5 seconds to 10 seconds – we don’t want our tests to timeout if the HTTP calls are too slow.

// jest/test.utils.js
import { Chromeless } from 'chromeless'

export const setup = () => {
  return new Chromeless(global.config.chromeless)

export const teardown = async chromeless => {
  try {
    await chromeless.end()
  } catch (err) {

And now, time for some tests!

We use the global Jest functions beforeAll and afterAll, to setup and teardown our test suites. The example below contains one regression test that checks the contents of our home page.

To do this, we visit our baseurl localhost:3000. Using the ‘evaluate’ method we can execute javascript code within Chrome, in the context of the DOM. This way, we can get the inner HTML of our main content node.

Using Jest snapshot testing, we can compare our snapshot with the previous one (which is stored in a separate snap file). if something changes, our test will fail.

// __tests__/index.test.js
beforeAll(() => { chromeless = setup() })
afterAll(async () => { await teardown(chromeless) })

test('+++ home renders correctly', async () => {
  const html = await chromeless
    .evaluate(() => document.querySelector('.content').innerHTML)

Before running a test, we need to make sure our static site is being served (using yarn start) and the URL is the same as the one defined in our config (ie. setup.default.js). If everything is up and running we can run yarn test. Notice that the first test run will generate a snapshot file. All subsequent tests will compare their snapshot to the stored one – if it differs, the test will fail. Feel free to modify the welcome text src/containers/Home.js and run your tests again.

// __tests__/index.test.js.snap
exports[`+++ home renders correctly`] = `"<h1>Welcome to React Static!</h1>"`;

Jest image snapshots

In the previous section, we wrote a regression test that checks if the HTML was as we were expecting. In order to create a visual regression test that compares image snapshots, we need an additional plugin: jest-image-snapshot. This plugin performs image comparisons using Blink-diff and behaves just like Jest snapshots do.

First, we need to install jest-image-snapshot:

    yarn add --dev jest-image-snapshot

Then, we add some functionality to our test utilities.

First, we extend Jest’s expect function, so we can use toMatchImageSnapshot() in our tests.

Second, we create a getFile function, which we will use to retrieve our generated screenshot.

// jest/test.utils.js
import { toMatchImageSnapshot } from 'jest-image-snapshot'
import * as fs from 'fs'

export const setup = () => {
  expect.extend({ toMatchImageSnapshot }) 

export const getFile = path => Promise.resolve(fs.readFileSync(path))

Now we have everything ready to create our new test! Below you can find an example where we use Chromeless to generate a screenshot of the page and we use Jest to expect that it matches our image snapshot.

// __tests__/index.test.js
test('+++ home renders correctly (visual)', async () => {
  const screenshotPath = await chromeless
  const screenshot = await getFile(screenshotPath)

Running yarn test will generate a screenshot and store that as an image snapshot. If we remove the navigation in app.js and run yarn test again, our test should fail and generate the following diff:

Running Chrome in headless mode

By default, Chromeless will start Chrome automatically in non-headless mode and will simply use the most recent version found on your system. You can override this behavior by starting Chrome yourself, and passing a flag of launchChrome: false in the Chromeless constructor.

To run our tests in headless mode, we need to follow these steps.

First, start by running Chrome in headless mode:

alias chrome="/Applications/Google\\ Chrome"
chrome --remote-debugging-port=9222 --disable-gpu --headless

Then create a new setup file (ie. setup.headless.js) and set launchChrome: false:

global.config = {
  chromeless: {
    launch: false,
  baseUrl: 'http://localhost:3000',

Third, add an additional script in the package.json file:

"test:headless": "jest --runInBand --setupFiles='<rootDir>/jest/setup.headless.js'"

And finally, you’re ready to run your test: yarn test:headless.

Running Chrome on AWS Lambda

As we explained earlier, it is possible to run Chrome in headless mode on AWS Lambda. This gives us the ability to speed up our tests by running them in parallel. Chromeless provides a proxy service that allows us to run and interact with Chrome remotely.

Before we can connect to the Chromeless proxy service we need to deploy it. You can follow these setup instructions – do make sure your region in the provider section is the same as your awsIotHost.

# serverless.yml
  awsIotHost: ''

  region: eu-central-1

When the Lambda function is deployed, we can add some functionality to our project to run our tests on Chrome remotely.

First, you start by creating a new setup file (ie. setup.remote.js) and set the correct remote options. Make sure your Lambda function has access to your base URL.

global.config = {
  chromeless: {
    remote: {
      endpointUrl: '',
      apiKey: 'YOUR_AWS_API_KEY',
  baseUrl: '',

Then, we add an additional test script in the package.json file for the remote setup. Notice that we remove the --runInBand option. As each Lambda function uses its own Chrome instance, we’re able to run our test suites in parallel.

"test:remote": "jest --setupFiles='<rootDir>/jest/setup.remote.js'"

Last but not least, we need to change our getFile function and add functionality so that it can download our screenshot from the remote:

const download = uri => new Promise((resolve, reject) => {
  request({ url: uri, method: 'GET', encoding: null },
    (error, response, body) => {
      if (!error && response.statusCode === 200) { resolve(body) } else { reject(error) }
export const getFile = path => (path.startsWith('https://') ? download(path) : Promise.resolve(fs.readFileSync(path)))

Finally, you can run your test: yarn test:remote.

Congratulations – you know how to do visual regression tests!

If you’d like more information, check out our github repo. Happy testing…

PS: did you fall a little in love with our repo? The feeling might be mutual! We’re looking for several more people to round out our dev squad – you can find all open vacancies here!