Skip to content

fix(jenkins): collect builds and stages that finish after the next sync - #9178

Open
pballester wants to merge 2 commits into
apache:mainfrom
pballester:fix/jenkins-stages-long-running-builds
Open

pballester wants to merge 2 commits into
apache:mainfrom
pballester:fix/jenkins-stages-long-running-builds

Conversation

@pballester

@pballester pballester commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes the incremental Jenkins collection described in #9177: builds that are still running during one sync and finish before the next are lost, entirely for single jobs and their stages for multi-branch jobs. Full diagnosis and evidence are in the issue. Two commits, kept separate for review:

1. fix(jenkins): collect stages of builds that finish after the previous sync

  • The build collectors store a build only once it has a result, so a build that started before the previous collection and finished after it reaches _tool_jenkins_builds in a later run. collectApiStages selected builds with tjb.start_time >= since, so it skipped those builds in that run and every run after, and their deployment stages never reached cicd_tasks / DORA.
  • Both stage collectors (single job and multi-branch) now select the builds that finished since the previous collection, through a small FinishedSince clause: tjb.timestamp + tjb.duration >= since.
  • timestamp (start) and duration are both milliseconds, so this is plain arithmetic, the same on MySQL and PostgreSQL. No dialect branching, and the filter stays in SQL.
  • It includes every build the old condition did (duration ≥ 0). The query is still scoped to the job, so the extra cost is a DB scan, not API calls.

2. fix(jenkins): re-collect single-job builds that were running at the last sync

  • For single jobs (not multi-branch), the build list is paged newest first and stops at the first build that started before the previous collection, and running builds were dropped. So a build running during one sync and finished before the next was never collected at all, neither the build nor its stages.
  • Filtering the list on finish time instead is not an option: the helper stops at the first older item, and finish times aren't ordered.
  • Running builds are now kept (stored with building = true), and the finalizable collector's CollectUnfinishedDetails re-collects them from job/<name>/<number>/api/json until they finish. This is the helper's intended pattern, as in pagerduty and circleci. A build deleted while running is skipped (404) instead of failing the task.
  • Multi-branch jobs are unchanged: they re-list every build on each run.
  • Side effect: single-job builds show up as in-progress cicd_pipelines (status IN_PROGRESS, result empty) while they run. The convertor already supports that, and DORA ignores them.

Upgrade note: incremental runs don't backfill what was already missed, because those builds finished before the new since. A full sync of the Jenkins scopes recovers them.

Tests

  • New e2e TestJenkinsStagesFinishedSince: builds finished before the last collection / started before and finished after / started after. It expects the last two; the old condition returns only the build that started after.
  • New e2e TestJenkinsUnfinishedBuilds (only this job's running builds are re-collected) and TestJenkinsUnfinishedBuildExtraction (a build listed while running, then re-collected finished, ends with its final state).
  • Jenkins e2e suite passes on MySQL 8.0 and PostgreSQL 14.
  • End to end against a real Jenkins (LTS 2.541.3) with a single pipeline job: a build waiting on an input during the first collection, finished before the second (incremental) one.
    • main: the build is missing from _tool_jenkins_builds, with no stages and no cicd_tasks.
    • This branch: the second run re-collects it from …/<number>/api/json, and it lands with building=0 result=SUCCESS, its stages and its cicd_tasks.
  • gofmt / go vet clean; golangci-lint reports no new findings in the plugin.

Does this close any open issues?

Closes #9177

Screenshots

N/A

… sync

Builds are stored only once they have a result, so a build that started
before the previous collection and finished after it reaches
_tool_jenkins_builds in a later run. The incremental stage collector filtered
on start_time >= since and skipped it forever.

Select builds that finished since the previous collection instead:
timestamp + duration (both milliseconds) >= since, which works the same on
MySQL and PostgreSQL and includes every build the old condition did.

Closes apache#9177
…ast sync

The single-job build collector lists builds newest first and stops at the
first one that started before the previous collection, and it dropped builds
that were still running. A build running during one sync and finished before
the next was therefore never collected: neither the build nor its stages.

Keep running builds (stored with building=true) and use the finalizable
collector's CollectUnfinishedDetails to re-collect them from
job/<name>/<number>/api/json until they finish. A build deleted while running
is skipped (404) instead of failing the task. Multi-branch jobs are not
affected: they re-list every build on each run.

Refs apache#9177
@pballester pballester changed the title fix(jenkins): collect stages of builds that finish after the previous sync fix(jenkins): collect builds and stages that finish after the next sync Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][jenkins] Incremental collection skips the stages of builds that finish after the next sync

1 participant