feat(workflows): route first implementations to Opus, regenerations to Sonnet - #11947
Merged
Merged
Conversation
…o Sonnet
impl-generate.yml now resolves the model per (spec, library) pair: an
explicit model input wins; otherwise ("auto", or the label trigger) the
pair's first implementation (no implementation file on origin/main yet)
runs on Opus and a regeneration on Sonnet. The checkout moves ahead of
"Extract inputs" so the routing can read origin/main, and every consumer
(both Claude steps, the metadata generated_by, the review dispatch, the
retry dispatch) keeps reading the one resolved steps.inputs.outputs.model.
The model input on impl-generate, bulk-generate and daily-regen defaults
to "auto" (choice lists keep haiku/sonnet/opus); bulk-generate and
daily-regen forward "auto" instead of an explicit sonnet, so scheduled
regens and bulk runs follow the routing. The babysit backfill scripts
default to auto as well. daily-regen's spec polish and similarity audit
move from Haiku to Sonnet.
Docs: pipeline models section in docs/workflows/overview.md, models
bullets in agentic/docs/project-guide.md, babysit-pipeline SKILL.md.
Tests run the real routing script against a throwaway repo.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv
The main flow threads impl-generate's resolved model through review and repair, but the rescue paths dispatch without one: impl-review-retry.yml and the watchdog's review rescues (`-f pr_number` only), the watchdog's repair rescues, and manual reruns. Those fell back to a hardcoded sonnet, so a first-run PR rescued mid-pipeline left Opus. impl-review.yml and impl-repair.yml now default their model input to "auto" and, when neither an input nor the repository_dispatch payload names a model, apply impl-generate's routing: opus while the pair has no implementation on origin/main, sonnet for a regeneration. impl-review resolves inside "Extract PR info" (matching the library file in any language directory, since LANGUAGE/EXT come one step later) so its four consumers stay unchanged; impl-repair gets a "Resolve model" step and its consumers read steps.model.outputs.model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv
… without origin/main Review follow-ups for the model routing: - impl-generate records the resolved model in the PR body (`**Model:** opus`, after the File line so it stays clear of #11945's insertion after the Parent Issue block). impl-review resolves input > payload > body marker > routing; impl-repair's "Resolve model" reads the marker via `gh pr view` when its input is auto and routes only when the PR has none. An explicit pin now survives rescues. - Routing no longer guesses a first run when origin/main is missing: it tries `git fetch origin +refs/heads/main:refs/remotes/origin/main` first; impl-generate then fails with an error, review and repair warn and assume opus. - The resolved model and its reason go to impl-generate's run summary and the issue preview comment ("Model: opus (first implementation)"); MODEL_SOURCE becomes MODEL_REASON with plain reasons. - bulk-generate paces dispatches 180 s apart for model auto/opus (120 s for sonnet/haiku); pace_seconds defaults to empty so the model decides. - Comment on the regex-safety of the impl-review ls-tree lookup; babysit SKILL.md warns that a gap backfill is all-Opus (slots=1 or a longer STAGGER) and flags the Sonnet-era timing numbers. - Tests: body marker vs. routing/input/payload for review and repair, a PR-body round trip from "Create Pull Request" into a rescue review, missing origin/main for all three workflows, summary + reason, pacing, and spec-create/report-validate/util-claude staying pinned to Opus. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv
Owner decision: a forced regeneration skips the regen gate and takes the fresh-generation path (repair loop, cascading thresholds), so without an explicit model it runs on Opus like a first implementation. "Extract inputs" gets the same REGEN_GATE expression as "Check for existing implementation" and, when the pair is already on origin/main and regen_gate=false, resolves opus with reason "forced regeneration". An explicit model still wins; a first run keeps its own reason. Lands here because #11945 (which added regen_gate) merged first; the branch is rebased onto it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Repair rescues can lose explicit model pins during API failures, and the queue’s new all-Opus default conflicts with its safe concurrency guidance.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Routes first implementations to Opus and regenerations to Sonnet while preserving explicit model pins across the pipeline.
Changes:
- Adds automatic model resolution and propagation.
- Updates pacing, rescue behavior, tests, and documentation.
- Changes daily preflight tasks from Haiku to Sonnet.
| File | Description |
|---|---|
.github/workflows/impl-generate.yml |
Resolves and records the model. |
.github/workflows/impl-review.yml |
Recovers pins and routes rescues. |
.github/workflows/impl-repair.yml |
Resolves models for repair runs. |
.github/workflows/bulk-generate.yml |
Adds model-aware pacing. |
.github/workflows/daily-regen.yml |
Uses automatic routing and Sonnet preflight. |
.claude/skills/babysit-pipeline/SKILL.md |
Documents routing and capacity guidance. |
.claude/skills/babysit-pipeline/run_spec.sh |
Documents the automatic model option. |
.claude/skills/babysit-pipeline/run_queue.sh |
Defaults queued backfills to automatic routing. |
tests/unit/workflows/test_model_routing.py |
Tests routing and workflow wiring. |
docs/workflows/overview.md |
Documents pipeline model selection. |
agentic/docs/project-guide.md |
Updates internal workflow guidance. |
changelog.d/model-routing.md |
Records the workflow changes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…r body lookup Copilot review on #11947: - run_queue.sh: under MODEL=auto (or opus) a gap backfill is all-Opus, yet the unattended queue still defaulted to 2 slots and a 90 s stagger. It now defaults to 1 slot / 180 s for auto and opus, and keeps 2 / 90 s for a pinned sonnet or haiku; the slots argument and STAGGER still override. SKILL.md's slot guidance and the example command (now `... <run> 1`) match. - impl-repair "Resolve model": a failed `gh pr view` used to pass for "no Model line" and silently reroute a pinned model. The lookup now retries like impl-review (3 attempts, 5 s / 10 s backoff) and fails the step with an error once they are spent; "Handle repair failure" then retries the attempt with model auto, which looks again. Tests: the queue defaults and overrides via run_queue.sh's library mode; the repair lookup with a fake gh that fails twice then succeeds, one that always fails (3 calls, error, no model output), and an explicit model that never calls gh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjHc4vXdEfWzM8G3FoYtGv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
origin/mainyet, now runs on Opus for generate, review and every repair. Regenerations stay on Sonnet. A forced regeneration (regen_gate=false) runs on Opus. An explicitmodelinput always overrides the routing.impl-generate.ymlresolves the model once, in "Extract inputs"; the checkout moves ahead of that step so it can readorigin/main. Every consumer reads that value: both Claude steps, the metadatagenerated_by, the review dispatch and the retry.**Model:** opus.origin/main. Iforigin/maincan't be read, impl-generate fails, and review and repair warn and assume a first run (Opus).auto. Themodelinput on impl-generate, bulk-generate, daily-regen, impl-review and impl-repair defaults toauto. bulk-generate and daily-regen forwardauto, so scheduled regens, bulk runs, label triggers and the babysit backfill scripts (MODEL=auto) all follow the routing. A full-catalogue backfill now runs on Opus.autooropus, and 120 s for a pinnedsonnetorhaiku.pace_secondsstill overrides.Plan
Routing test.
git cat-file -e origin/main:plots/$SPEC/implementations/$LANG/$LIB$EXT, the same "implementation file alone" test that the regen gate (feat(workflows): regen gate — regenerations replace a live implementation only when visibly better #11945) uses. impl-review matches the library file in any language directory, because LANGUAGE and EXT are set one step later.Model order. impl-review: input, then repository_dispatch payload, then the PR body
**Model:**line, then routing. impl-repair: input, then the PR body line, then routing.Docs.
docs/workflows/overview.md: a "Pipeline models" section.agentic/docs/project-guide.md: a models block and the new pacing..claude/skills/babysit-pipeline/SKILL.md: the routing, the all-Opus backfill guidance (slots=1or a longerSTAGGER) and a note that the timing numbers were measured on Sonnet.changelog.d/model-routing.md.Review. An independent review of the design and the implementation came back ship-with-fixes. The fixes are applied:
origin/main;The owner approved routing forced regenerations to Opus.
Test plan
uv run --extra test --extra dev pytest tests/unit -q: 2228 passed. The newtests/unit/workflows/test_model_routing.pyruns the real step scripts against a throwaway repo with a fakeorigin/mainand a fakegh. It covers:.R,.jl,.jsand.tsx;origin/mainin all three workflows;ruff check,ruff format --check, actionlint on the changed workflows,tools.changelog check --base origin/main**Model:** opusin the PR body;