Why we like stopOnFailure in our test suite

Recently I was working on a client project, slowly making it a little better one PR at a time, and in this particular PR I overrode the phpunit.xml default and set stopOnFailure="true".

The reviewer was curious why I made the change, and I thought it would make a good tip to share here with everyone.

When I'm running the test suite locally, and there's a failure, I just want the test run to stop, so I can fix it. I don't want to have to wait for it all to finish then scroll back up through a bunch of errors to find the first one to fix.

Knowing there is one failure or four failures doesn't really mean much. Either way, I'm going to fix one of the failures at a time and then re-run the test suite.

And if this is running in CI, I still want it to fail fast. Why waste resources running the full test suite when the developer is going to need to fix the first failure and re-run the suite locally anyway?

Plus, sometimes a single broken line of code can result in a cascade of failures. Seeing 35 failed tests might paint an incorrect picture if I have a typo in one method used throughout the app. Fixing that one failure would actually fix all the failures.

So running all the tests in that case results in more noise and more failures that might not actually matter. More test failures doesn't necessarily mean more bugs to fix.

And even if there are two legitimately different errors, I'd still argue you should only fix one at a time, with a test run in between. Anytime we try to do multiple things at once, it opens us up to making mistakes.

Here's my workflow:

• Fix the bug, run the failing test.
• If it passes, run the whole test class that the failing test was in.
• If that passes, run the whole folder of tests that the failing test was in.
• If that passes, run the whole suite.

Making it easy to quickly run tests of different scopes is a good practice to get into.

All that said, there is one time I can think of where I do like to see all the failures at once, and that's when I first join a project and there's a broken test suite.

In that specific case, it can be useful to know if it's 1 broken test or 100. But for that one-off event, I can just temporarily tweak phpunit.xml for a single test run to see all the failures.

Here to help,

Joel

P.S. Do you have a broken test suite? Would you like to make testing an enjoyable part of your development process? Let's talk.