Safer testing of records that should not exist

Any time I run into a flaky test, I like to step back and see if there's a way to avoid the same issue in the future.

Recently, I found a test that consistently failed when I tried to run just the Feature test suite. But if I ran the whole test suite or all the tests in that class, it consistently passed.

The test was on a validation rule that required an ID in the request to match an existing record in the database for the client. My test was failing because the fake record ID I provided, 999, was actually being found in the database, so the rule was passing when I expected it to fail.

Why was this happening? I'm using the RefreshDatabase trait for my feature tests, which wraps each test in a database transaction. But even though the records are rolled back at the end of each test, MySQL's auto-increment value keeps marching forward.

So when I run the test in isolation, and the 5 default records are created, the IDs would be 1 through 5, and 999 would not be found. And if I run the whole test suite, the IDs created by default in the test would be much higher, well over 999, so no collision.

But it was only in this one weird situation running just the Feature test suite where it happened to create the 5 default records with IDs ranging from 998 to 1003, causing the collision. Super random and super annoying.

What is the solution? When picking a fake ID, make it much higher, like 9999999. Sure, if my test suite grows super huge, it could fail again, but it's not likely. I'm happy with this solution, but I'd love to hear if you have a different approach. Hit reply and let me know.

Here to help,

Joel

P.S. Do you have a flaky test suite? I have a great track record of fixing them. Don't live with the pain. Let's fix it.