Everything here comes from the CI of public repositories and is rebuilt every night. Nothing is seeded. Each test execution went through the same API, worker and scoring as any other Flakemetry instance. Last built 2026-11-18 22:18 UTC.
Web tests against SauceDemo on Chromium, Firefox and WebKit.
Oct 5, 2026 to Nov 18, 2026. Repository
API tests against a hosted demo platform and a local copy of it in Docker.
Aug 6, 2026 to Nov 18, 2026. Repository
The same test cases on Appium for Android and iOS, XCUITest and Espresso.
Aug 6, 2026 to Nov 18, 2026. Repository
The test suite of this tool, against a real database.
Nov 4, 2026 to Nov 18, 2026. Repository
Playwright e-commerce suite runs on three browsers, and each browser is scored on its own, so a failure on WebKit alone reads as a failure there and not as a flake everywhere. Open a test from suite health, then a trace to see its steps.
No test scores above 0.10.
Restful Booker API suite tests a hosted demo it does not run. In late October that demo started answering 500 on every per-id route, and the suite's readiness gate stopped the run before any test started. Those runs appear as no tests ran rather than as gaps, and no test is marked down for them: 14 runs on 10 nights so far.
A partial outage is set aside too. On Aug 31 the hosted platform failed 22 of the 128 tests that ran. 19 of them failed with the same error and passed on that commit on other nights, so those failures do not count against them. The other 3 failed in their own way and still count, so they stay on the board with that night as their last flake.
The top of that board is below. The live and local variants are scored
apart, because a local copy says nothing about the hosted one.
| Score | Test | Why |
|---|---|---|
| 0.28 | function-level authorization (BFLA) @security > denies message.delete to a caller with an invalid token without changing the target live | SAME_SHA_VARIANCE |
| 0.24 | function-level authorization (BFLA) @security > denies booking.update to a caller with an invalid token without changing the target live | SAME_SHA_VARIANCE |
| 0.24 | function-level authorization (BFLA) @security > denies booking.delete to a caller with an invalid token without changing the target live | SAME_SHA_VARIANCE |
| 0.20 | booking properties @property > rejects any second booking overlapping the same window live | SAME_SHA_VARIANCE |
| 0.15 | cross-service consistency @contract > a booking created via BookingService is reflected in the room report live | SAME_SHA_VARIANCE |
Mobile task manager suite runs the same catalogue of test cases through Appium on Android and iOS, XCUITest and Espresso. The same app and the same cases make the runners directly comparable. Retry settings differ, with up to three iterations on XCUITest, so read the failed attempts alongside the retries.
The job status alone is misleading. XCUITest went red on 2 of 82 runs and looks like the most reliable of the four, yet on 21 of the runs it reported green a test had failed and only a retry saved it. Nobody would have seen those without per-attempt history.
| appium-android | appium-ios | xcuitest | espresso | |
|---|---|---|---|---|
| Runs | 84 | 85 | 82 | 82 |
| Went red | 16 | 9 | 2 | 11 |
| of which no test ran | 11 | 4 | 2 | 5 |
| of which most of the suite failed | 2 | 0 | 0 | 2 |
| Cancelled before it finished | 1 | 2 | 2 | 0 |
| Green only because a retry rescued it | 0 | 4 | 21 | 0 |
| Test cases seen | 64 | 64 | 63 | 60 |
| Failed attempts in the other runs | 0 | 9 | 24 | 4 |
| Tests that passed on retry | 0 | 9 | 24 | 0 |
Red and green are the job's own conclusion in CI, so a job can go red with every recorded test passing, for example when it could not start a session for the rest. A cancelled job was stopped before it finished, so it is neither: the tests it got through are kept, and it does not count as red.
The test cases with the most trouble. Runs where most of the suite failed together are left out, because they say nothing about one test. A red cell is a failed attempt, and "retried" means the same test then passed.
| Test case | appium-android | appium-ios | xcuitest | espresso |
|---|---|---|---|---|
| TC-TASK-018 | 68 clean | 80 clean | 4 failed, 4 retried | 1 failed |
| TC-FILTER-002 | 71 clean | 82 clean | 3 failed, 3 retried | 1 failed |
| TC-FILTER-004 | 71 clean | 1 failed, 1 retried | 2 failed, 2 retried | 75 clean |
| TC-SEARCH-006 | 70 clean | 1 failed, 1 retried | 2 failed, 2 retried | 75 clean |
| TC-SEARCH-002 | 71 clean | 82 clean | 2 failed, 2 retried | 75 clean |
| TC-TASK-003 | 68 clean | 80 clean | 2 failed, 2 retried | 75 clean |
| TC-TASK-022 | 70 clean | 1 failed, 1 retried | 1 failed, 1 retried | 75 clean |
| TC-AUTH-004 | 68 clean | 77 clean | 1 failed, 1 retried | 75 clean |
| TC-FILTER-003 | 71 clean | 82 clean | 1 failed, 1 retried | 75 clean |
| TC-FILTER-005 | 71 clean | 81 clean | 1 failed, 1 retried | 75 clean |
| TC-FILTER-007 | 71 clean | 81 clean | 1 failed, 1 retried | 75 clean |
| TC-NAV-001 | 12 clean | 1 failed, 1 retried | 9 clean | 12 clean |
| TC-SEARCH-003 | 70 clean | 81 clean | 1 failed, 1 retried | 75 clean |
| TC-TASK-001 | 68 clean | 80 clean | 1 failed, 1 retried | 75 clean |
| TC-TASK-007 | 68 clean | 80 clean | 1 failed, 1 retried | 75 clean |
| TC-TASK-014 | 68 clean | 80 clean | 1 failed, 1 retried | 75 clean |
| TC-TASK-020 | 70 clean | 1 failed, 1 retried | 79 clean | 75 clean |
| TC-TASK-021 | 70 clean | 1 failed, 1 retried | 79 clean | 75 clean |
| TC-TASK-023 | 70 clean | 1 failed, 1 retried | 78 clean | 75 clean |
| TC-TASK-024 | 70 clean | 1 failed, 1 retried | 78 clean | 75 clean |
| TC-TASK-025 | 70 clean | 1 failed, 1 retried | 78 clean | 75 clean |
| TC-AUTH-012 | 68 clean | 80 clean | 79 clean | 1 failed |
| TC-FILTER-001 | 71 clean | 82 clean | 78 clean | 1 failed |
| Score | Test | Why |
|---|---|---|
| 0.34 | TC-NAV-001 — Tapping a task card navigates to Task Details appium-ios | SAME_SHA_VARIANCE, PASS_ON_RERUN |
| 0.33 | test_TC_TASK_018_confirmingDeleteRemovesTask xcuitest | SAME_SHA_VARIANCE, PASS_ON_RERUN |
| 0.33 | test_TC_SEARCH_002_partialTitleMatch xcuitest | SAME_SHA_VARIANCE, PASS_ON_RERUN, FAIL_ISOLATION |
| 0.27 | test_TC_TASK_003_newTaskAppearsInList xcuitest | SAME_SHA_VARIANCE, PASS_ON_RERUN |
| 0.26 | test_TC_FILTER_002_statusDoneShowsOnlyCompletedTasks xcuitest | SAME_SHA_VARIANCE, PASS_ON_RERUN, FAIL_ISOLATION |
Flakemetry itself runs its whole suite every day against a real database and goes on the same board, under the same rules.
No test scores above 0.10.
live runs of Restful Booker API suite and the Espresso runs of Mobile task manager suite log how many tests passed and name only the failures, so the other tests are taken as passed. Failures, skips and the XCUITest and Appium attempts are named in the logs.main branch is included.Run it on your own suite · How this page is built · Flakemetry on GitHub