Engineering Council Test Reliability Report

Scope aligned with Slack channel #dezvoltare, covering 2026-02-18 to 2026-03-19. Metrics and timings are sourced from GitLab pipelines, jobs, and test-report artifacts for the daily 6 PM regression suite and the production smoke suite. Trend charts use weekly buckets across this window.

Executive Snapshot

29
Daily Runs
11/29
Daily Green
23m 08s
Avg Daily Runtime
71
Smoke Attempts
52/71
Smoke Green
3m 17s
Avg Smoke Runtime
3m 04s
Median Smoke Time
0
Current Green Streak

Council Narrative

  • Daily suite reliability finished strong: the last 0 daily runs were all green.
  • In the last 10 daily runs, total test volume grew from 1,177 to 1,188.
  • Library has the highest strict failure ratio at 0.87%, while Billing has the broadest non-pass footprint at 1.75% and showed up in 10 failed runs.
  • The average daily-suite runtime, measured from GitLab start and finish timestamps, was 23m 08s.
  • Smoke runs stayed fast when healthy: average duration was 3m 17s and the median passing run was 3m 04s.
  • 4 production pipelines clearly recovered on rerun after an initial smoke failure: 145497, 145743, 147232, 147447.
  • 5 failed smoke attempts never reached test execution in GitLab; they are counted as failed attempts, with per-test counts marked unavailable.

Category Failure Ratios

How computed

Failure Ratio = failed test executions divided by total test executions in the selected timeframe.

Non-pass Ratio = (failed + pending + skipped) test executions divided by total test executions in the selected timeframe.

Example: a 1.75% Billing failure ratio means 1.75% of all Billing test executions in this period ended in failed. A 1.35% Library non-pass ratio means 1.35% of all Library test executions in this period ended in failed, pending, or skipped.

How computed

Failure Ratio = failed test executions divided by total test executions in the selected timeframe.

Non-pass Ratio = (failed + pending + skipped) test executions divided by total test executions in the selected timeframe.

Example: a 1.75% Billing failure ratio means 1.75% of all Billing test executions in this period ended in failed. A 1.35% Library non-pass ratio means 1.35% of all Library test executions in this period ended in failed, pending, or skipped.

Category Failure RatioBilling1.75%Web0.28%Frontend0.40%Library0.87%
Category Non-pass Ratio (fail + pending + skipped)Billing1.75%Web0.34%Frontend0.52%Library1.35%
Weekly Daily Suite Status0135702-1602-2303-0203-0903-16
Weekly Smoke Attempts0612182402-1602-2303-0203-0903-16
Weekly Average Daily Suite Runtime17m 52s22m 47s27m 41s32m 35s37m 30s02-1602-2303-0203-0903-16
Weekly Average Smoke Runtime3m 03s3m 23s3m 44s4m 05s4m 26s02-1602-2303-0203-0903-16
Daily Suite Total Test Growth (Recent 10 Runs)1177117911821185118803-0903-1103-1303-1503-1703-18

Category Aggregate Table

How computed

Failure Ratio = failed test executions divided by total test executions in the selected timeframe.

Non-pass Ratio = (failed + pending + skipped) test executions divided by total test executions in the selected timeframe.

Example: a 1.75% Billing failure ratio means 1.75% of all Billing test executions in this period ended in failed. A 1.35% Library non-pass ratio means 1.35% of all Library test executions in this period ended in failed, pending, or skipped.

How computed

Failure Ratio = failed test executions divided by total test executions in the selected timeframe.

Non-pass Ratio = (failed + pending + skipped) test executions divided by total test executions in the selected timeframe.

Example: a 1.75% Billing failure ratio means 1.75% of all Billing test executions in this period ended in failed. A 1.35% Library non-pass ratio means 1.35% of all Library test executions in this period ended in failed, pending, or skipped.

CategoryTotalFailedPendingSkippedFailure RatioNon-pass RatioRuns With Failures
Billing262746001.75%1.75%10
Web1263336070.28%0.34%7
Frontend478319600.40%0.52%7
Library148613070.87%1.35%4
Billing
Total 2627Runs with failures 10
Failed46
Pending0
Skipped0
Runs with failures10
Failure Ratio1.75%
Non-pass Ratio1.75%
Web
Total 12633Runs with failures 7
Failed36
Pending0
Skipped7
Runs with failures7
Failure Ratio0.28%
Non-pass Ratio0.34%
Frontend
Total 4783Runs with failures 7
Failed19
Pending6
Skipped0
Runs with failures7
Failure Ratio0.40%
Non-pass Ratio0.52%
Library
Total 1486Runs with failures 4
Failed13
Pending0
Skipped7
Runs with failures4
Failure Ratio0.87%
Non-pass Ratio1.35%

Recent Runs

Recent Daily Suite Runs

DatePipelineStatusSummary
2026-03-11 18:21148293PASSEDTotal 1183 | Passed 1183 | Failed 0
2026-03-12 18:22148396PASSEDTotal 1188 | Passed 1188 | Failed 0
2026-03-13 18:21148495PASSEDTotal 1188 | Passed 1188 | Failed 0
2026-03-14 18:22148497PASSEDTotal 1188 | Passed 1188 | Failed 0
2026-03-15 18:23148499PASSEDTotal 1188 | Passed 1188 | Failed 0
2026-03-16 18:23148604FAILEDTotal 1188 | Passed 1187 | Failed 1
2026-03-17 18:22148823PASSEDTotal 1188 | Passed 1188 | Failed 0
2026-03-18 18:22149034FAILEDTotal 1188 | Passed 1187 | Failed 1
2026-03-11 18:21
PASSED
Pipeline 148293
Total1183
Passed1183
Failed0
Pending0
2026-03-12 18:22
PASSED
Pipeline 148396
Total1188
Passed1188
Failed0
Pending0
2026-03-13 18:21
PASSED
Pipeline 148495
Total1188
Passed1188
Failed0
Pending0
2026-03-14 18:22
PASSED
Pipeline 148497
Total1188
Passed1188
Failed0
Pending0
2026-03-15 18:23
PASSED
Pipeline 148499
Total1188
Passed1188
Failed0
Pending0
2026-03-16 18:23
FAILED
Pipeline 148604
Total1188
Passed1187
Failed1
Pending0
2026-03-17 18:22
PASSED
Pipeline 148823
Total1188
Passed1188
Failed0
Pending0
2026-03-18 18:22
FAILED
Pipeline 149034
Total1188
Passed1187
Failed1
Pending0

Recent Smoke Attempts

DatePipelineStatusPassedFailedDuration
2026-03-16 22:08148635
job
PASSED11002m 58s
2026-03-16 22:38148640
job
PASSED11003m 00s
2026-03-16 22:59148644
job
PASSED11002m 57s
2026-03-16 23:53148646
job
PASSED11003m 00s
2026-03-17 00:06148648
job
PASSED11003m 03s
2026-03-17 00:24148651
job
PASSED11003m 03s
2026-03-17 12:06148704
job
PASSED11003m 04s
2026-03-17 16:09148791
job
PASSED11003m 04s
2026-03-17 16:56148817
job
PASSED11003m 09s
2026-03-18 16:49149024
job
PASSED11003m 10s
2026-03-16 22:08
PASSED
Pipeline 148635
job
Passed110
Failed0
Duration2m 58s
2026-03-16 22:38
PASSED
Pipeline 148640
job
Passed110
Failed0
Duration3m 00s
2026-03-16 22:59
PASSED
Pipeline 148644
job
Passed110
Failed0
Duration2m 57s
2026-03-16 23:53
PASSED
Pipeline 148646
job
Passed110
Failed0
Duration3m 00s
2026-03-17 00:06
PASSED
Pipeline 148648
job
Passed110
Failed0
Duration3m 03s
2026-03-17 00:24
PASSED
Pipeline 148651
job
Passed110
Failed0
Duration3m 03s
2026-03-17 12:06
PASSED
Pipeline 148704
job
Passed110
Failed0
Duration3m 04s
2026-03-17 16:09
PASSED
Pipeline 148791
job
Passed110
Failed0
Duration3m 04s
2026-03-17 16:56
PASSED
Pipeline 148817
job
Passed110
Failed0
Duration3m 09s
2026-03-18 16:49
PASSED
Pipeline 149024
job
Passed110
Failed0
Duration3m 10s
Generated from GitLab project adservio/helm2. Times are shown in Europe/Bucharest. Daily-suite runtime is measured from GitLab pipeline and job timestamps. Category counts come from GitLab test-report JSON artifacts, with job-trace fallback when older artifacts have expired.