The state of testing at Firmhouse and what to improve

We are big on automated software testing at Firmhouse.

• We write tests to confirm new functionality of our software works.
• We also write tests so that nothing we built in the past breaks with new updates.
• And we write tests so that bugs we made in the past won't be re-introduced.

There's nothing sadder than a customer hitting a bug again after you fixed it a few weeks ago.

In addition to writing software tests for the functionality of our platform (more on that below). We also have a bunch of tools in place that'll make sure we write sane code, and that we won't introduce any malicious code ourselves or depend on third party plugins that contain security leaks.

All of the above is done automated. Every time one of our developers pushes a commit to their branch on GitHub, the whole test suite and automated checks run. And after a developer merges their Pull Request into our main branch, all the tests are run again!

Here are some of the details on what we use:

• We use GitHub to host our code and collaborate using GitHub Pull Requests.
• On every commit pushed CircleCI runs our automated test suite.
• In the build stage on CircleCI:
  - our assets are compiled;
  - brakeman checks for malicious code or security flaws introduced;
  - improved-yarn-audit and bundle-audit checks for security holes in our dependencies;
  - and Rubocop and ESLint verify if everyone is conforming to our shared code standards.
• Then our three test suites are run in parallel: unit tests, system tests, and concurrency tests. Within these suites we also apply parallelisation with test timing to speed up the automated test run as a whole.
• We use minitest to write our tests. And we use context/should blocks wherever applicable, and regular test definitions in older parts of our codebase.
• We're making more and more use of ViewComponents to build our UI. So that also makes testing frontend easier, without having to rely on slow system tests.
• A full build runs in approximately 7-8 minutes.

There's a couple of things we're improving as time goes by:

• Some older parts of our platform are slightly under-tested, while most other parts are heavily over-tested. The latter unfortunately causing the test suite to be quite slow compared to its coverage. We can probably improve there.
• We have some brittle tests sometimes popping up. Usually having to do with our paralellization approach where one test already flushes the DB or file fixes, while a test in progress still relies on that data. It's very hard to fix. Seems there are a few features in maxitest that might be able to help us.
• To speed things up parallelize even more. Or switch from CircleCI to Buildkite or a different CI runner.
• Less system tests. More unit tests on the UI. System tests are by far the slowest part of our test run. And a lot of system tests are unnecessary or include too much for what the test should be about. Heavily trimming those system tests will probably yield a lot faster build while not impacting effective coverage.

Overall, I'm quite happy with what we have. Our development and deployment pipeline is very nice to work with. We're shipping multiple times per day. In recent months we've never shipped something broken that could have been avoided by writing more automated tests.