feat: auto-trigger PR review when all readiness criteria are met#323
Conversation
Fires a review-agent dispatch when a PR meets all readiness criteria: - All CI checks passing - No CHANGES_REQUESTED reviews - No unresolved review threads Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds a reusable GitHub Actions workflow that validates PR readiness (open, not draft, checks passing, no requested changes, no unresolved threads) and conditionally dispatches a ChangesPR Auto-Review Readiness Gate
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new GitHub Actions workflow, pr-auto-review.yml, which serves as a caller for a reusable PR readiness check. The review feedback identifies that the workflow triggers are incomplete, missing essential pull_request events like opened and ready_for_review. Additionally, the job permissions are currently insufficient and need to be expanded to include read access for checks, statuses, and actions to ensure the automation can correctly retrieve CI status and metadata.
| on: | ||
| check_suite: | ||
| types: [completed] | ||
| pull_request_review: | ||
| types: [submitted, dismissed] | ||
| pull_request_review_thread: | ||
| types: [resolved] |
There was a problem hiding this comment.
The workflow is intended to trigger when a PR becomes ready for review, but the current trigger set misses several critical events. Specifically, it should include the pull_request event with types opened, reopened, and ready_for_review.
ready_for_review: Essential for triggering the check when a draft PR is marked as ready.opened/reopened: Ensures the workflow runs for PRs that might already satisfy all readiness criteria upon creation (e.g., if CI was already completed on the branch before the PR was opened).
Without these, the automation will not fire in these common scenarios, requiring manual intervention which defeats the purpose of the auto-trigger.
on:
check_suite:
types: [completed]
pull_request:
types: [opened, reopened, ready_for_review]
pull_request_review:
types: [submitted, dismissed]
pull_request_review_thread:
types: [resolved]| permissions: | ||
| pull-requests: read |
There was a problem hiding this comment.
The current permissions are insufficient for the readiness criteria described in the PR summary. The gh pr checks command (and the underlying GitHub API calls) requires checks: read and statuses: read to retrieve the status of CI runs. Additionally, actions: read is recommended to allow the reusable workflow to inspect other workflow runs, aligning with the permissions granted to other review-related agents in the organization (e.g., Claude Code as documented in standards/ci-standards.md).
permissions:
pull-requests: read
checks: read
statuses: read
actions: read![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/pr-auto-review-reusable.yml:
- Around line 121-129: The script incorrectly counts historical
CHANGES_REQUESTED reviews by using the gh api call that lists all reviews and
setting CHANGES_REQ, causing false positives; replace that logic with a call to
gh pr view to read the effective reviewDecision field and base the check on
reviewDecision == "CHANGES_REQUESTED" (or not "APPROVED") instead of the
CHANGES_REQ count, updating the section that references CHANGES_REQ and the gh
api "/repos/${REPO}/pulls/${PR_NUMBER}/reviews" call to use gh pr view --json
reviewDecision and branch the logic on the reviewDecision value.
In `@standards/workflows/pr-auto-review.yml`:
- Around line 31-32: The workflow uses an invalid GitHub Actions event name
"pull_request_review_thread"; update the workflow trigger to a supported event
such as "pull_request_review" (or "pull_request_review_comment" if you need
comment-specific threads) by replacing the "pull_request_review_thread" entry in
the pull_request_review_thread trigger block with the correct event name so
actionlint stops reporting "unknown Webhook event." Ensure you update the types
array if needed to match the chosen event's valid types.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7ddbb5a0-75ba-4fce-bab7-a5636499c045
📒 Files selected for processing (3)
.github/workflows/pr-auto-review-reusable.yml.github/workflows/pr-auto-review.ymlstandards/workflows/pr-auto-review.yml
There was a problem hiding this comment.
Pull request overview
Adds an org-level PR auto-review workflow that dispatches the existing review agent when a pull request appears ready without requiring a manual bot mention.
Changes:
- Adds a reusable readiness-gate workflow that resolves the PR, checks CI/review/thread criteria, and dispatches the review agent.
- Adds a local thin caller workflow for this repository.
- Adds a standard caller template for adoption by other org repositories.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.github/workflows/pr-auto-review-reusable.yml |
Implements readiness checks and repository dispatch logic. |
.github/workflows/pr-auto-review.yml |
Adds this repo’s local caller for the reusable workflow. |
standards/workflows/pr-auto-review.yml |
Adds the reusable workflow adoption template for other repos. |
Comments suppressed due to low confidence (2)
.github/workflows/pr-auto-review-reusable.yml:123
- Counting every historical
CHANGES_REQUESTEDreview means a reviewer who later submits an approving/comment-only review still leaves the old changes-requested review in this list, so the auto-review can remain permanently blocked after the request is resolved. Derive the latest non-dismissed review state per reviewer (or use the review decision from GraphQL) instead of counting all historical reviews.
# 3. No reviews in CHANGES_REQUESTED state.
CHANGES_REQ=$(gh api "/repos/${REPO}/pulls/${PR_NUMBER}/reviews" \
--jq '[.[] | select(.state == "CHANGES_REQUESTED")] | length')
.github/workflows/pr-auto-review-reusable.yml:136
- This only inspects the first 100 review threads. PRs with more than 100 threads can have an unresolved thread beyond the first page and still dispatch the review agent; paginate
reviewThreadsuntilhasNextPageis false or query an aggregate that accounts for all threads.
reviewThreads(first:100){nodes{isResolved}}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| name: PR Auto-Review — Ready Check | ||
|
|
||
| on: | ||
| check_suite: |
| name: PR Auto-Review — Ready Check | ||
|
|
||
| on: | ||
| check_suite: |
| # 2. All CI checks must be completed and passing. | ||
| # gh pr checks --json returns objects with a "bucket" field: | ||
| # "pass" | "skipping" | "fail" | "pending" | "cancel" | ||
| CHECKS=$(gh pr checks "$PR_URL" --json bucket,name 2>/dev/null || echo "[]") |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8543b1be5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| NOT_PASSING=$(echo "$CHECKS" | \ | ||
| jq '[.[] | select(.bucket != "pass" and .bucket != "skipping")] | length') |
There was a problem hiding this comment.
When this workflow is running for a ready PR, it creates its own PR check before this step executes; gh pr checks reports pending checks in the pending bucket, and this filter treats any non-pass/non-skipping bucket as not ready. That means the auto-review run blocks itself on every check_suite, review, or thread-resolution trigger and never reaches the dispatch step unless its own check is excluded or only the intended required CI checks are queried.
Useful? React with 👍 / 👎.
| check_suite: | ||
| types: [completed] |
There was a problem hiding this comment.
The stated CI-finished trigger will not run for the org's normal GitHub Actions checks: GitHub's Actions docs for check_suite say the event does not trigger workflows when the suite was created by GitHub Actions or the head SHA is associated with GitHub Actions. For PRs whose readiness depends on these workflows, no auto-review run is started when CI turns green, so this needs a trigger that actually fires for Actions CI completion.
Useful? React with 👍 / 👎.
| set -euo pipefail | ||
|
|
||
| # Fetch basic PR metadata in one call. | ||
| PR_META=$(gh pr view "$PR_URL" --json state,isDraft,number,baseRepository) |
There was a problem hiding this comment.
gh pr view --help/the GitHub CLI manual lists the supported JSON fields and baseRepository is not one of them, so this command exits with an unknown-field error before any readiness criteria are checked. As written, every non-skipped trigger fails here instead of dispatching; use a supported field such as url/headRepository or derive the base repo from the PR URL/API response.
Useful? React with 👍 / 👎.
- Replace pull_request_review_thread (invalid GH Actions event) with workflow_run and pull_request event handling - Add workflow_run:completed trigger for Actions-based CI (check_suite does not fire for GH Actions runs) - Add pull_request:[opened,reopened,synchronize,ready_for_review] trigger - Use reviewDecision from gh pr view instead of REST reviews list to correctly reflect effective review state (avoids false positives from historical CHANGES_REQUESTED reviews) - Fix gh pr checks capture: use || true instead of || echo '[]' so the JSON payload is preserved when checks are failing/pending - Exclude own in-progress check run from CI gate to prevent self-blocking - Add checks:read and actions:read job permissions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove pull_request_review_thread (invalid GH Actions event) - Add workflow_run:[CI] for Actions-based CI completion - Add pull_request:[opened,reopened,synchronize,ready_for_review] - Add checks:read and actions:read job permissions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove pull_request_review_thread (invalid GH Actions event) - Add workflow_run with TODO comment for CI workflow name customization - Add pull_request:[opened,reopened,synchronize,ready_for_review] - Add checks:read and actions:read job permissions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add '# shellcheck disable=SC2016' before jq --arg self usage; the $self token is a jq variable (passed via --arg), not a shell variable - Replace baseRepository (not a valid gh pr view --json field) with URL parsing: sed 's|https://github.com/||; s|/pull/.*||' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace single-quoted jq filter containing $self (a jq variable) with a double-quoted string using \\, which produces a literal dollar sign for jq without triggering shellcheck SC2016 (expressions don't expand in single quotes). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GraphQL variables ($owner, $repo, $number) in the -f query= string triggered SC2016. Use a double-quoted string with \\$ escaping so the shell produces literal dollar signs for GraphQL without shellcheck interpreting them as unexpanded shell variables in single quotes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…herit Resolves SonarCloud security hotspot S7635 (githubactions:S7635). Only GH_PAT_WORKFLOWS is needed by the reusable workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addressing CodeRabbit CHANGES_REQUESTEDBoth issues from CodeRabbit's review have been fixed in earlier commits:
Also just pushed: replaced @coderabbitai please re-review. |
|
Kicking off a fresh review now! ✅ Actions performedFull review triggered. |
|
don-petry
dismissed
coderabbitai[bot]’s stale review
Both issues addressed: reviewDecision used instead of historical reviews list; pull_request_review_thread removed. Dismissing to unblock merge.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c829492edf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # to catch CI turning green on a PR. | ||
| # TODO: replace "CI" with your repository's CI workflow name(s). | ||
| workflow_run: | ||
| workflows: ["CI"] |
There was a problem hiding this comment.
With the standard workflow set I checked (Dependency audit, AgentShield, and this repo's SonarCloud Analysis are also GitHub Actions workflows), gh pr checks gates on checks beyond CI. If CI completes while one of those checks is still pending, this run skips; when that later Actions workflow turns green, this workflow_run filter does not fire and check_suite still will not cover GitHub Actions-created suites, so ready PRs can remain undispatched until some unrelated PR/review event. Include every Actions workflow that can appear in the readiness gate, or trigger on all completed workflow runs and let the existing gate decide readiness.
Useful? React with 👍 / 👎.
| pull_request_review: | ||
| types: [submitted, dismissed] |
There was a problem hiding this comment.
When the only remaining blocker is an unresolved review thread, resolving the final thread does not submit or dismiss a PR review, so this workflow never re-runs to observe UNRESOLVED=0; GitHub exposes thread resolution as separate pull_request_review_thread webhook activity rather than as pull_request_review. In that scenario the PR satisfies all criteria but no auto-review is dispatched until another unrelated PR or CI event occurs, so add a supported mechanism to re-evaluate on thread resolution (for example a webhook/dispatch bridge or scheduled fallback).
Useful? React with 👍 / 👎.
| pull_request: | ||
| types: [opened, reopened, synchronize, ready_for_review] |
There was a problem hiding this comment.
For Dependabot PRs, GitHub treats pull_request/pull_request_review workflows as fork-originated and does not expose normal Actions secrets, so this trigger can invoke the reusable without GH_PAT_WORKFLOWS; the next gh pr view call then fails authentication instead of cleanly skipping. Because this organization relies on Dependabot workflows, opening or updating a Dependabot PR will produce a red auto-review workflow run unless the event is skipped before the PAT is needed or the credential is also supplied as a Dependabot secret.
Useful? React with 👍 / 👎.
| UNRESOLVED=$(gh api graphql \ | ||
| -f "query=query(\$owner:String!,\$repo:String!,\$number:Int!){repository(owner:\$owner,name:\$repo){pullRequest(number:\$number){reviewThreads(first:100){nodes{isResolved}}}}}" \ | ||
| -f owner="${REPO%%/*}" \ | ||
| -f repo="${REPO##*/}" \ | ||
| -F number="${PR_NUMBER}" \ | ||
| --jq "[.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false)] | length") |
There was a problem hiding this comment.
When a large PR has more than 100 review threads and the first page is resolved but a later page still has an unresolved thread, this query only inspects reviewThreads(first:100) and then dispatches the review agent even though the stated readiness criterion is not met. Please loop on pageInfo.hasNextPage/endCursor or otherwise query all review threads before treating UNRESOLVED=0 as authoritative.
Useful? React with 👍 / 👎.
| pull_request: | ||
| types: [opened, reopened, synchronize, ready_for_review] |
There was a problem hiding this comment.
On same-repository PRs, the pull_request run has access to normal Actions secrets while executing the workflow definition from the PR merge ref, so a branch that modifies this stub or the referenced reusable can read GH_PAT_WORKFLOWS before the change is merged. Because this token is described as a classic repo-scoped PAT, use a default-branch context such as pull_request_target/workflow_run for the secret-bearing dispatch path or split the untrusted PR trigger from the privileged API call.
Useful? React with 👍 / 👎.
Summary
Adds a new
pr-auto-reviewworkflow that automatically dispatches the review agent when a PR becomes ready for review — no manual@donpetry-botmention needed.Trigger events
check_suite: completedpull_request_review: submitted/dismissedpull_request_review_thread: resolvedReadiness criteria (all must pass)
gh pr checks)CHANGES_REQUESTEDreviews — checked via the Reviews REST APIreviewThreads.isResolved)If any criterion fails the workflow exits silently with no dispatch.
Files
.github/workflows/pr-auto-review-reusable.yml.github/workflows/pr-auto-review.ymlstandards/workflows/pr-auto-review.yml@v2)Adoption
Copy
standards/workflows/pr-auto-review.ymlto.github/workflows/pr-auto-review.ymlin any repo. TheGH_PAT_WORKFLOWSorg secret is already present everywhere.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Summary by CodeRabbit