This is the fifth and final post in the series. If you haven’t read the previous posts, I recommend doing so for added context:
I omitted a lot of information about Sharp Cooking’s testing in the previous posts in this series. I wanted to write a dedicated post about the experience, and this is it.
Unit testing vs. end-to-end testing
I am somewhat biased when it comes to unit testing Single Page Applications (SPAs). I have had very little success in the projects I worked on before. As the codebase grows, the tests have become brittle, and the value has diminished. To be fair, I did put a token effort into unit testing Sharp Cooking, but I ended up discarding the few tests I had.
On the other hand, I have had great success with end-to-end (e2e) testing for SPAs. The key to e2e testing is to test interactions from a user perspective. For example, I expect users to create recipes manually as well as import them from a website, so my test steps should mimic what a user would do and validate the result in a similar manner.
Given my bias and the information above, I have decided to focus on e2e tests in the Sharp Cooking SPA app.
Playwright is a cross-platform, cross-browser, and cross-language testing framework from Microsoft. For Sharp Cooking, I used Playwright tests written in TypeScript and executed them in Chrome, Safari, and Firefox for desktop and Chrome and Safari for mobile. This configuration covers the most common devices I expect Sharp Cooking will be used on. Note that the mobile tests are executed in emulation mode and not on a real device.
I have used Nightwatch.js and Smart Bear Test Complete to perform e2e tests before. Compared to them, I found Playwright’s setup, including CI/CD, much easier, and the tooling much more complete.
Create the first few tests
Using Playwright, you can either manually create tests or use the codegen utility. The codegen is a great way to get started quickly. If you have written e2e tests, you know that constructing
XPath queries is not fun at all. The first thing you will notice is that the codegen doesn’t actually suggest
XPath. As you hover over components, the most common suggested way to identify them is via
To start the codegen tool:
Here is a short gif of how it works:
Quite a few of the tests currently in the Sharp Cooking codebase were created via codegen. All they needed was a bit of code cleanup.
As the HTML tree becomes deep or large, it is a good idea to leverage test ids. They work as a marker for Playwright to find particular elements. After creating one, the codegen will favor getByTestId() over other approaches. The code below demonstrates the creation of a test id directly in HTML:
A real test would look like this:
A bit of complication
Wouldn’t it be great if all tests were simple like what we’ve seen this far? Wouldn’t it be great…
Some user interactions require a bit more work than others. For instance, Sharp Cooking uses the browser-fs-access to open and save files. While I typically prefer not to mock anything, it is unavoidable in this situation.
To mock the browser-fs-access file picker, we need to use a
page.addInitScript() that replaces
window.showOpenFilePicker with a version that always returns a particular blob file handle. Note that the fileHandle is a shallow mock that returns a specific file, created off a blob with known JSON content. Maybe the actual code can explain it better:
Sharp Cooking makes a few calls to its API. In particular, zip backups, image processing, and import from URL make API calls. However, the API endpoints are already tested separately, so I decided not to test them again. Instead, the tests use
page.route to intercept and return results we can control for the tests:
Oh Safari, why?
As usual, Safari doesn’t play like the others. Some tests simply do not work in Safari. For instance, any test that mocks the API calls fails in Safari. I have spent way too much time trying to fix those but nothing works, so for now, they are skipped:
Another challenge with testing in Safari is related to timeouts. Often, tests will fail because the default 30 seconds timeout will expire. There are 2 approaches to improving this:
- Increase the timeout in each affected test:
- Allow for retries in the playwright.config.ts file:
retries: process.env.CI ? 2 : 0
Technically, if a test fails once and passes next it is considered flaky. I recommend you put as much effort in fixing the actual root cause, but sometimes that is easier said than done so having a retry is not entirely a bad thing.
Another practice that may lead to flakiness is random input. As tempting as it is to have a generator create your field values, it can create issues as each execution will use different values. It is a bit too easy to get it wrong so I typically avoid it entirely.
E2e tests can take a while to run. As of writing, Sharp Cooking runs 162 total tests and takes between 5 and 10 minutes to execute. GitHub Actions have an agent timeout of 1 hour that needs to be considered. A similar timeout exists for Azure DevOps Pipelines as well.
That’s a close
As I mentioned before, this will be the last post in this series. I expect to write much more about Sharp Cooking in the future, but as far as describing the thought process in each decision I think I covered enough. If you have any specific question, please drop a comment here or in Sharp Cooking’s Git Hub.
I had a lot of fun working on Sharp Cooking new app as well as writing these articles.