flakemetry

Real test suites, real history

Everything here comes from the CI of public repositories and is rebuilt every night. Nothing is seeded. Each test execution went through the same API, worker and scoring as any other Flakemetry instance. Last built 2026-11-18 22:18 UTC.

Playwright e-commerce suite

Web tests against SauceDemo on Chromium, Firefox and WebKit.

478
runs
4,157
test executions
0
failed attempts
0
passed on retry

Oct 5, 2026 to Nov 18, 2026. Repository

Restful Booker API suite

API tests against a hosted demo platform and a local copy of it in Docker.

263
runs
35,856
test executions
34
failed attempts
0
passed on retry

Aug 6, 2026 to Nov 18, 2026. Repository

Mobile task manager suite

The same test cases on Appium for Android and iOS, XCUITest and Espresso.

333
runs
15,462
test executions
203
failed attempts
33
passed on retry

Aug 6, 2026 to Nov 18, 2026. Repository

Flakemetry itself

The test suite of this tool, against a real database.

16
runs
20,673
test executions
0
failed attempts
0
passed on retry

Nov 4, 2026 to Nov 18, 2026. Repository

A healthy suite says so

Playwright e-commerce suite runs on three browsers, and each browser is scored on its own, so a failure on WebKit alone reads as a failure there and not as a flake everywhere. Open a test from suite health, then a trace to see its steps.

No test scores above 0.10.

A run that never started is not a flake

Restful Booker API suite tests a hosted demo it does not run. In late October that demo started answering 500 on every per-id route, and the suite's readiness gate stopped the run before any test started. Those runs appear as no tests ran rather than as gaps, and no test is marked down for them: 14 runs on 10 nights so far.

A partial outage is set aside too. On Aug 31 the hosted platform failed 22 of the 128 tests that ran. 19 of them failed with the same error and passed on that commit on other nights, so those failures do not count against them. The other 3 failed in their own way and still count, so they stay on the board with that night as their last flake.

The top of that board is below. The live and local variants are scored apart, because a local copy says nothing about the hosted one.

ScoreTestWhy
0.28 function-level authorization (BFLA) @security > denies message.delete to a caller with an invalid token without changing the target live SAME_SHA_VARIANCE
0.24 function-level authorization (BFLA) @security > denies booking.update to a caller with an invalid token without changing the target live SAME_SHA_VARIANCE
0.24 function-level authorization (BFLA) @security > denies booking.delete to a caller with an invalid token without changing the target live SAME_SHA_VARIANCE
0.20 booking properties @property > rejects any second booking overlapping the same window live SAME_SHA_VARIANCE
0.15 cross-service consistency @contract > a booking created via BookingService is reflected in the room report live SAME_SHA_VARIANCE

One app, four runners

Mobile task manager suite runs the same catalogue of test cases through Appium on Android and iOS, XCUITest and Espresso. The same app and the same cases make the runners directly comparable. Retry settings differ, with up to three iterations on XCUITest, so read the failed attempts alongside the retries.

The job status alone is misleading. XCUITest went red on 2 of 82 runs and looks like the most reliable of the four, yet on 21 of the runs it reported green a test had failed and only a retry saved it. Nobody would have seen those without per-attempt history.

appium-androidappium-iosxcuitestespresso
Runs84858282
Went red169211
of which no test ran11425
of which most of the suite failed2002
Cancelled before it finished1220
Green only because a retry rescued it04210
Test cases seen64646360
Failed attempts in the other runs09244
Tests that passed on retry09240

Red and green are the job's own conclusion in CI, so a job can go red with every recorded test passing, for example when it could not start a session for the rest. A cancelled job was stopped before it finished, so it is neither: the tests it got through are kept, and it does not count as red.

The test cases with the most trouble. Runs where most of the suite failed together are left out, because they say nothing about one test. A red cell is a failed attempt, and "retried" means the same test then passed.

Test caseappium-androidappium-iosxcuitestespresso
TC-TASK-01868 clean80 clean4 failed, 4 retried1 failed
TC-FILTER-00271 clean82 clean3 failed, 3 retried1 failed
TC-FILTER-00471 clean1 failed, 1 retried2 failed, 2 retried75 clean
TC-SEARCH-00670 clean1 failed, 1 retried2 failed, 2 retried75 clean
TC-SEARCH-00271 clean82 clean2 failed, 2 retried75 clean
TC-TASK-00368 clean80 clean2 failed, 2 retried75 clean
TC-TASK-02270 clean1 failed, 1 retried1 failed, 1 retried75 clean
TC-AUTH-00468 clean77 clean1 failed, 1 retried75 clean
TC-FILTER-00371 clean82 clean1 failed, 1 retried75 clean
TC-FILTER-00571 clean81 clean1 failed, 1 retried75 clean
TC-FILTER-00771 clean81 clean1 failed, 1 retried75 clean
TC-NAV-00112 clean1 failed, 1 retried9 clean12 clean
TC-SEARCH-00370 clean81 clean1 failed, 1 retried75 clean
TC-TASK-00168 clean80 clean1 failed, 1 retried75 clean
TC-TASK-00768 clean80 clean1 failed, 1 retried75 clean
TC-TASK-01468 clean80 clean1 failed, 1 retried75 clean
TC-TASK-02070 clean1 failed, 1 retried79 clean75 clean
TC-TASK-02170 clean1 failed, 1 retried79 clean75 clean
TC-TASK-02370 clean1 failed, 1 retried78 clean75 clean
TC-TASK-02470 clean1 failed, 1 retried78 clean75 clean
TC-TASK-02570 clean1 failed, 1 retried78 clean75 clean
TC-AUTH-01268 clean80 clean79 clean1 failed
TC-FILTER-00171 clean82 clean78 clean1 failed

The top of the board

ScoreTestWhy
0.34 TC-NAV-001 — Tapping a task card navigates to Task Details appium-ios SAME_SHA_VARIANCE, PASS_ON_RERUN
0.33 test_TC_TASK_018_confirmingDeleteRemovesTask xcuitest SAME_SHA_VARIANCE, PASS_ON_RERUN
0.33 test_TC_SEARCH_002_partialTitleMatch xcuitest SAME_SHA_VARIANCE, PASS_ON_RERUN, FAIL_ISOLATION
0.27 test_TC_TASK_003_newTaskAppearsInList xcuitest SAME_SHA_VARIANCE, PASS_ON_RERUN
0.26 test_TC_FILTER_002_statusDoneShowsOnlyCompletedTasks xcuitest SAME_SHA_VARIANCE, PASS_ON_RERUN, FAIL_ISOLATION

Its own tests too

Flakemetry itself runs its whole suite every day against a real database and goes on the same board, under the same rules.

No test scores above 0.10.

What you are looking at

Run it on your own suite · How this page is built · Flakemetry on GitHub