In this article we'll be collecting the best practices of dealing with flaky tests.

First we start from a set of methods you can apply to flaky tests without trying to fix them.

  1. Launch a test up to X times until first success.
  2. If X times from Y runs passed - consider the test passed.
  3. Quarantine (exclude from future runs) if probability of pass and failure is more than X percent after Y runs.
  4.  If status is the same as during previous run - ignore results.
  5.  Run and ignore status - gather statistics for future decision.
  6. Throw away. Test itself may be badly implemented.

There are different reasons for test flakiness and various techniques of fixing it. Below you can find links to some interesting articles about this topic.