feat(workflows): regen gate — regenerations replace a live implementation only when visibly better by MarkusNeusinger · Pull Request #11945 · MarkusNeusinger/anyplot

MarkusNeusinger · 2026-09-27T09:04:48Z

Summary

A regeneration (an implementation PR for a pair that already has an implementation on origin/main) now gets one review and no repair loop. The review scores the new render blind, re-scores the predecessor's production renders against the same criteria, and writes review_regen.json (step 8b of ai-quality-review.md). The stored score is never shown to the reviewer, neither in the prompt nor in the previous-review file; it appears only in the gate's notice line and comment.
automation/scripts/regen_gate.py merges only when:
- the new score is at least the re-scored predecessor minus 1;
- at least one improvement names where it is visible;
- nothing regressed. On a *-basic spec, a replaced data scenario or added encodings count as regressions unless a change request asked for them.
Anything missing or malformed keeps the live implementation. Predictable format slips are coerced and named in the reason.
Merge adds regen:improved, then ai-approved. Keep adds regen:kept, closes the PR with a comment, and restores impl:<lib>:done. Keep never adds ai-rejected, so the repair loop's delete-from-main path cannot run on a regen.
Escape hatch: regen_gate=false on impl-generate.yml / bulk-generate.yml labels the PR regen:forced and uses today's fresh-generation path.
Watchdog:
- re-closes open regen:kept PRs (Case 0);
- skips regens in the repair cases;
- accepts the regen markers in Case 5;
- warns on a regen stranded without ai-approved.
daily-regen ages specs by the newer of their metadata updated and their spec issue's last activity, so a kept spec isn't re-picked the next night. Any issue activity postpones that spec's regen.
Other changes:
- A failed regeneration no longer sets impl:<lib>:failed.
- The local /regen withholds ai-approved below the stored score minus 1.

Background: a 5-round regen experiment on bubble-basic. Across it, 39 of 75 regens failed their first review, were repaired, and merged at the lowered bar, replacing better implementations (e.g. 93 → 88). The review never saw the predecessor. Review scores moved ±4 on near-identical renders.

Heads-up for merge time: right after this merges, in-flight implementation PRs whose head lacks the new review prompt will fail closed to keep. They are detected as regenerations from origin/main, but their reviewer writes no review_regen.json. Merge when no implementation/* PRs are open, or expect those to close as regen:kept. daily-regen.yml is disabled until this PR and the follow-up criteria PR are merged.

Plan

Implements PR 1 of the regen-gate plan (agentic/runs/regen-exp-bubble-basic/PLAN.md, local), with the owner-approved changes from an independent review:

the predecessor is re-scored in the same session;
W/P/C ids replace verbatim weakness matching;
regeneration is detected from origin/main;
label order is regen:improved before ai-approved;
keep bookkeeping;
the daily-regen pick fix;
the local ratchet;
the escape hatch;
the notice line;
the watchdog changes;
the failure-handler changes.

The follow-up PR adds the per-spec "What a good version looks like" section and the criteria changes.

Decisions:

Helper scripts run from the workflow's own ref: they are copied to $RUNNER_TEMP before the PR head is checked out.
regen:forced PRs do not carry regen, so they keep the repair cases.
An invalid id invalidates the whole review_regen.json.
The predecessor renders live at /tmp/anyplot-prev-plot-{light,dark}.png, because the review's Read allowlist covers /tmp/* only.
When dispatched from a branch, prompts/ is overlaid from the trigger ref (the pre-merge test hook) and unstaged, so it cannot enter the metadata commit.

Test plan

uv run pytest tests/unit -q (2059 passed). This covers the regen_gate decision table and coercions, close_issue_if_complete, and the workflow content guards: no ai-rejected on regen, ai-approved re-added before the impl-merge dispatch, detection from origin/main, and no stored score in the prompt.
ruff check, ruff format --check, mypy api core
actionlint on the six changed workflows. With shellcheck, only pre-existing findings and SC2086 info (house style).
uv run python -m tools.changelog check --base origin/main
Independent review before push (ship-with-fixes; fixes applied in 404fb1a)
Smoke test after merge on bubble-basic:
- ggplot2 should end as regen:improved if the invisible size legend gets fixed and named.
- lets-plot should end as regen:kept: PR closed, main and production GCS untouched, no sync-postgres run, pending removed, watchdog quiet.
Workflow and prompt changes have no local verification loop. They are only observable on real pipeline runs.

Residual risks:

Forced regens that exhaust their repairs still delete the live implementation (existing behavior, documented).
A regen review that crashes twice ends in manual attention.
The 1000-issue listing is sorted by most recent update. Only issues that have been quiet the longest can fall outside it, and such a spec simply falls back to metadata updated.
P ids are format-checked only.

A regeneration (implementation already on origin/main) gets one review and no repair. The review scores the new render blind, re-scores the predecessor's production renders against the same criteria and writes review_regen.json (ai-quality-review.md step 5f). regen_gate.py merges only when new >= prev_rescored - 1, at least one improvement is visible and nothing regressed (-basic: scenario swap / added encodings count unless a change request asked for them); everything missing or malformed keeps. - impl-review: detect regen from origin/main, regen context with stable weakness ids, download previous renders, gate before any verdict label, regen:improved before ai-approved, keep closes the PR, restores impl:<lib>:done and never adds ai-rejected - impl-generate: regen_gate input (regen:forced escape hatch), regen label and PR body context, no impl:<lib>:failed on a failed regen, review on the trigger ref when dispatched from a branch - watchdog: Case 0 re-closes regen:kept, repair cases skip regens, Case 5 accepts regen markers; daily-regen counts kept attempts; bulk-generate passthrough; impl-merge shares close_issue_if_complete.py - local /regen ratchet withholds ai-approved below stored - 1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv

…d, pick window - impl-review: the regen merge branch re-asserts ai-approved before the impl-merge dispatch; the watchdog flags regen:improved without ai-approved instead of treating it as a verdict - blind scoring: PREVIOUS_SCORE is gone from the review prompt, the previous review is extracted without stored scores (--omit-scores), the before/after body moved to step 8b after the score caps, and the predecessor renders live at /tmp/anyplot-prev-plot-{light,dark}.png (outside the working dir; /tmp is the Read allowlist), named only in 8b - daily-regen: the attempt record is the spec issue's updatedAt (one gh issue list, most recently updated first, limit 1000) instead of a 200-PR window that dropped kept specs after ~13 nights - regen_gate: coerce predictable slips (digit-string prev_rescored, lower-case refs, missing lists, "n/a" change_request_applied) and name each in the reason; neutralise @-mentions in the comment summary - prompt: JSON parse self-check, attempt line no longer says /3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv

Copilot

Copilot review overview

🟡 Changes recommended

Regeneration detection can fail on early setup errors, predecessor scores leak into re-scoring, and some scheduling and watchdog behavior contradicts the documented contract.

Review effort: Balanced
Findings: 3 · 3

Open (6)

Specs without metadata bypass minimum-age checks · New
Failure handler loses regeneration state after setup failures · New
Watchdog retries failed regenerations beyond the retry limit · New
Failure message uses plural noun for a single library · New
Stored score leaks into supposedly independent rescoring · New
Module docstring references the wrong contract step · New

What changed in this PR

Adds a regeneration gate that replaces live plot implementations only when a blind comparison finds visible improvement without regressions.

Changes:

Adds gate logic, workflow routing, watchdog handling, and an escape hatch.
Updates daily regeneration scheduling and local regeneration safeguards.
Adds documentation and comprehensive unit/content-guard tests.

File	Description
`automation/scripts/regen_gate.py`	Implements regeneration decisions and review extraction.
`automation/scripts/close_issue_if_complete.py`	Centralizes issue-completion bookkeeping.
`.github/workflows/impl-review.yml`	Integrates comparison review and gate verdicts.
`.github/workflows/impl-generate.yml`	Detects, labels, and retries regenerations.
`.github/workflows/impl-merge.yml`	Uses shared completion logic and reports prior scores.
`.github/workflows/bulk-generate.yml`	Adds the gate-bypass input.
`.github/workflows/daily-regen.yml`	Considers issue activity when selecting specs.
`.github/workflows/watchdog-stuck-jobs.yml`	Handles gated regeneration states.
`prompts/workflow-prompts/ai-quality-review.md`	Defines blind predecessor comparison output.
`agentic/workflows/modules/regen/pr_create.py`	Adds the local score ratchet.
`agentic/commands/regen.md`	Documents local override behavior.
`.claude/skills/babysit-pipeline/SKILL.md`	Documents regeneration monitoring.
`docs/workflows/overview.md`	Documents the gate and labels.
`agentic/docs/project-guide.md`	Updates internal workflow guidance.
`automation/scripts/README.md`	Documents the new scripts.
`changelog.d/regen-gate.md`	Adds the changelog fragment.
`tests/unit/automation/scripts/test_regen_gate.py`	Tests gate decisions and CLI behavior.
`tests/unit/automation/scripts/test_close_issue_if_complete.py`	Tests completion bookkeeping.
`tests/unit/workflows/test_regen_gate_workflows.py`	Guards workflow invariants.
`tests/unit/agentic/regen/test_pr_create.py`	Tests the local score ratchet.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

codecov · 2026-09-27T09:12:54Z

Codecov Report

❌ Patch coverage is 88.67925% with 48 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
automation/scripts/close_issue_if_complete.py	53.73%	31 Missing ⚠️
automation/scripts/regen_gate.py	94.94%	17 Missing ⚠️

📢 Thoughts on this report? Let us know!

- daily-regen: a spec without usable metadata is aged by its issue activity too (MIN_AGE applies); datetime.min only when neither exists. Covered by a test that runs the pick script against a fake gh. - impl-generate: the failure handler re-detects a live implementation from origin/main itself, so a setup step failing before "Check for existing implementation" never marks a live pair impl:<lib>:failed. - watchdog: Case 1 no longer re-dispatches a failed regen review (it warns); regen detection also checks the implementation file on main unless regen:forced. - impl-review: the predecessor source copy has its generated "Quality: N/100" header hidden (regen_gate.py sanitize-source). - close_issue_if_complete: singular "library" for one failure. - test docstring points at step 8b. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv

# Conflicts: # automation/scripts/README.md

…11946) ## Summary - **A new spec section.** Specs gain a closing `## What a good version looks like` section: 3-5 observable properties of a good render of that plot type, including "expected, not a defect" items. The template and spec-create (first attempt and retry, one line per bullet) write it for new specs. Spec polish must never add or edit it. - **A "Plot-Type Characteristics" subsection in the quality criteria.** - A property the section names is never a weakness, and removing one deducts. - When a spec has no section, the review infers the characteristics from Description, Data and Notes. - The section describes the plot type, so it cannot waive auto-reject, theme readability, the canvas contract, VQ-07 or VQ-01. - **Soft rules from the bubble-basic regen experiment:** - VQ-02: data-mark overlap counts only when it hides information. - VQ-03: legend glyphs must be visible in both themes. - SC-03: marks sit at their data values. Post-hoc displacement deducts in proportion. Exempt: categorical jitter, layout-positioned types, and jitter, dodge or offset the spec asks for. - SC-01: on a `-basic` spec, an encoding the spec neither requires nor offers as optional is the wrong variant and earns no DE-03 or LM-02 credit. - SC-02: only properties a good version *shows* count as features; "expected" statements are permissions. - **The review prompt mirrors these rules** in `ai-quality-review.md`: step 1, step 5d, the step-7 table, and "Important". - **Generation rules for a regen:** - Address only the weaknesses that are real per the section, and name the declined ones in the commit body. - Answer overlap via data, marker size or alpha, never by moving marks. - Keep the scenario and the variant. - Add no code that doesn't show in the render. - The same rules reach impl-repair and plot-generator. - **One-off hand edit (owner-approved).** The section is seeded by hand in 8 specs: bubble-basic, scatter-basic, line-basic, bar-basic, heatmap-basic, heatmap-correlation, violin-basic and network-force-directed. - Hand-editing `plots/*/specification.md` normally goes through the pipeline. - These edits only append the section. Description, Applications, Data, Notes and `specification.yaml` (including `updated`) are unchanged. - Merging triggers the normal sync-postgres run, whose parsed fields don't change, and an IndexNow ping for those 8 spec pages. Background: in a 5-round regen experiment on bubble-basic, the review applied generic ideals such as "overlapping bubbles are a weakness". Regens answered that by moving marks off their data values. They also added encodings to a basic spec, and an invisible ggplot2 size legend was never flagged in six rounds. ## Plan - **Placement.** The section sits last, so the Notes split in `sync_to_postgres.py` can't capture it. The review uses it; it is not synced to the database. - **Format.** A `## ` heading plus column-0 `- ` bullets, no sub-headings. - **Length.** The prompts ask for one line per bullet. The format test allows indented continuation lines and 2-6 bullets, so house style never blocks a spec-create PR. - **Reach.** Prompt changes reach only implementation PRs created after the merge, because impl-review reads prompts from the PR head. Follow-up with #11945 (regen gate), in whichever PR merges second: - `regen_gate.parse_characteristics` appends indented continuation lines to the preceding bullet. - The "jitter" regression wording in step 8b adopts the SC-03 exemptions. Process: an independent review of the plan and of this implementation came back ship-with-fixes. All findings are applied in 48d75e6. ## Test plan - [x] `uv run --extra test --extra dev pytest tests/unit -q`: 2046 passed. New tests cover: - the template section and the criteria and review-prompt rules; - the generate, repair and plot-generator no-displacement rule and the polish prohibition; - the Notes boundary in sync_to_postgres; - the parser format for every spec that has the section; - both spec-create prompts requesting the section one line per bullet. - [x] `ruff check`, `ruff format --check`, `mypy api core`, `actionlint` on spec-create.yml, `tools.changelog check` - [ ] No local check exists for `prompts/` or `.github/workflows/`. Verify on real runs: - the next spec-create writes the section; - the next spec polish leaves it alone; - a bubble-basic regen doesn't move marks and flags invisible legend glyphs. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…o Sonnet (#11947) ## Summary - **Routing.** A (spec, library) pair's first implementation, with no implementation file on `origin/main` yet, now runs on Opus for generate, review and every repair. Regenerations stay on Sonnet. A forced regeneration (`regen_gate=false`) runs on Opus. An explicit `model` input always overrides the routing. - **One resolved value.** `impl-generate.yml` resolves the model once, in "Extract inputs"; the checkout moves ahead of that step so it can read `origin/main`. Every consumer reads that value: both Claude steps, the metadata `generated_by`, the review dispatch and the retry. - **Visible model.** The run summary and the issue preview comment show the model and why, for example "opus (first implementation)". The PR body records it as `**Model:** opus`. - **Rescues keep a pin.** Review and repair rescues that arrive without a model (impl-review-retry, the watchdog, manual reruns) read that PR body line back, so an explicit pin survives them. Only a PR without the line is routed. These rescues used to fall back to a hardcoded Sonnet. - **No guessing without `origin/main`.** If `origin/main` can't be read, impl-generate fails, and review and repair warn and assume a first run (Opus). - **Default `auto`.** The `model` input on impl-generate, bulk-generate, daily-regen, impl-review and impl-repair defaults to `auto`. bulk-generate and daily-regen forward `auto`, so scheduled regens, bulk runs, label triggers and the babysit backfill scripts (`MODEL=auto`) all follow the routing. A full-catalogue backfill now runs on Opus. - **Pacing.** bulk-generate waits 180 s between dispatches for model `auto` or `opus`, and 120 s for a pinned `sonnet` or `haiku`. `pace_seconds` still overrides. - **Fixed models.** daily-regen's spec polish and cross-library similarity audit move from Haiku to Sonnet. spec-create, report-validate and util-claude stay on Opus, now guarded by a test. ## Plan - **Routing test.** `git cat-file -e origin/main:plots/$SPEC/implementations/$LANG/$LIB$EXT`, the same "implementation file alone" test that the regen gate (#11945) uses. impl-review matches the library file in any language directory, because LANGUAGE and EXT are set one step later. - **Model order.** impl-review: input, then repository_dispatch payload, then the PR body `**Model:**` line, then routing. impl-repair: input, then the PR body line, then routing. - **Docs.** - `docs/workflows/overview.md`: a "Pipeline models" section. - `agentic/docs/project-guide.md`: a models block and the new pacing. - `.claude/skills/babysit-pipeline/SKILL.md`: the routing, the all-Opus backfill guidance (`slots=1` or a longer `STAGGER`) and a note that the timing numbers were measured on Sonnet. - Changelog: `changelog.d/model-routing.md`. - **Review.** An independent review of the design and the implementation came back ship-with-fixes. The fixes are applied: - the PR-body model pin; - fail-closed handling without `origin/main`; - pacing; - a run summary; - guards on the workflows that stay on Opus. The owner approved routing forced regenerations to Opus. ## Test plan - [x] `uv run --extra test --extra dev pytest tests/unit -q`: 2228 passed. The new `tests/unit/workflows/test_model_routing.py` runs the real step scripts against a throwaway repo with a fake `origin/main` and a fake `gh`. It covers: - routing for auto, empty, label trigger, explicit override and forced regeneration, across `.R`, `.jl`, `.js` and `.tsx`; - the PR body line against routing, input and payload, for review and repair; - a round trip from "Create Pull Request" into a rescue review; - a missing `origin/main` in all three workflows; - content guards for wiring, pacing and the fixed Opus workflows. - [x] `ruff check`, `ruff format --check`, actionlint on the changed workflows, `tools.changelog check --base origin/main` - [ ] Workflow changes have no local verification loop. After merge, confirm on real runs: - "model: opus (first implementation)" on a fresh pair, together with `**Model:** opus` in the PR body; - "sonnet (regeneration)" on the next regen; - 180 s pacing in the bulk-generate preview. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

MarkusNeusinger and others added 3 commits September 27, 2026 10:50

docs: document the regen gate, labels and escape hatch

505df0d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv

Copilot AI balanced review requested due to automatic review settings September 27, 2026 09:04

docs(changelog): add PR reference to fragment

a9ec51f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv

Copilot started reviewing on behalf of MarkusNeusinger September 27, 2026 09:05 View session

Copilot AI reviewed Sep 27, 2026

View reviewed changes

MarkusNeusinger and others added 2 commits September 27, 2026 11:17

Merge remote-tracking branch 'origin/main' into feat/regen-gate

c13f138

# Conflicts: # automation/scripts/README.md

MarkusNeusinger mentioned this pull request Sep 27, 2026

feat(prompts): judge reviews against per-plot-type characteristics #11946

Merged

3 tasks

MarkusNeusinger merged commit 02e1a79 into main Sep 27, 2026
12 checks passed

MarkusNeusinger deleted the feat/regen-gate branch September 27, 2026 10:11

MarkusNeusinger mentioned this pull request Sep 27, 2026

feat(workflows): route first implementations to Opus, regenerations to Sonnet #11947

Merged

3 tasks

Navigation Menu

feat(workflows): regen gate — regenerations replace a live implementation only when visibly better - #11945

feat(workflows): regen gate — regenerations replace a live implementation only when visibly better#11945
MarkusNeusinger merged 6 commits into
mainfrom
feat/regen-gate

MarkusNeusinger commented Sep 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MarkusNeusinger commented Sep 27, 2026

Summary

Plan

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Sep 27, 2026 •

edited

Loading