Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the checkpoints.yaml file for the GitHub release skill, standardizing property names like target and pattern, updating script paths to vendor/bin/, and introducing new checks for reusable workflows and changelog link integrity. Feedback focuses on logic errors in the command patterns: the tag integrity check (GR-6) appears to have inverted exit logic that would cause false positives, and the jq filters for release assets (GR-9, GR-10) require the -e flag to correctly communicate success or failure via exit codes.
There was a problem hiding this comment.
Pull request overview
This PR updates the github-release skill’s checkpoint definitions to match the runner’s expected target/pattern schema (instead of legacy path/content/command), and adjusts a few release-related checks (notably cosign signature formats and workflow_dispatch severity) so checkpoints produce meaningful PASS/FAIL results.
Changes:
- Standardize checkpoint field names to
target/patternso mechanical checks execute correctly. - Broaden GR-10 to accept additional cosign artifact naming conventions, and demote GR-5 (workflow_dispatch) to
info. - Add new release-pipeline integrity checks (GR-12, GR-13) and adjust several command-based checks.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| skills/github-release/checkpoints.yaml | Converts legacy checkpoint fields to target/pattern, tweaks severities, expands cosign artifact handling, and adds new command-based integrity checks. |
| docs/superpowers/specs/2026-04-16-unified-eval-schema-design.md | Removed the unified eval schema design spec document. |
| docs/superpowers/plans/2026-04-16-unified-eval-schema.md | Removed the unified eval schema implementation plan document. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…n formats The runner expects field names target: and pattern: (with command: as alias for type=command). github-release was using path:/content:/command:, so the runner skipped every check with 'Target file not found:'. Convert all 13 checkpoints to the standard schema and they actually run again. Side fixes: - GR-1..GR-13 now run against repos that delegate to the netresearch skill-repo-skill release workflow, because the local release.yml grants id-token + attestations + tag triggers (which is exactly what the reusable workflow needs). Inline patterns that match the local file prove the local file is correctly wired into the reusable workflow. - GR-5 demoted to info: the upstream skill-repo-skill release workflow is intentionally tag-push-only because workflow_dispatch + auto-bump produced unsigned tags. Inline workflows that opt in are fine, but it's not required. - GR-6 reformulated without $() / ; / && (forbidden by the runner's command whitelist). Still asserts no lightweight tags. - GR-7 / GR-12 / GR-13 reference vendor/bin/ paths (the runner only whitelists vendor/bin/* — $CLAUDE_PLUGIN_ROOT/scripts is rejected). These checkpoints work when the validator scripts are installed there. - GR-10 (cosign signature) now accepts .bundle (modern Sigstore) OR .sig + .pem (the detached format actually produced by the netresearch skill-repo-skill release workflow). Signed-off-by: Sebastian Mendel <github@sebastianmendel.de>
…duplicates Address reviewer findings on the checkpoint schema standardization: - GR-6: wrap the lightweight-tag detector in `test -z "$(...)"` so the command exits 0 only when no offenders are found. The bare `grep -v` pattern inverted the meaning (exit 0 when offenders exist) and exited 1 on empty output (no tags at all). Now: annotated-only -> 0, any lightweight -> 1, empty repo -> 0. - GR-9: add `jq -e` so a missing SBOM asset produces a non-zero exit instead of printing the literal string "false" with exit 0, which the runner would treat as success. - GR-10: same `jq -e` fix, plus tighten the predicate to require either a `.bundle` OR the pair (`.sig` AND `.pem`). The previous regex `\\.(bundle|sig|pem)$` accepted a lone `.sig` or lone `.pem`, which is not a valid cosign signature set. - Drop the legacy old-schema GR-12/GR-13 entries that re-appeared after rebasing onto main (main introduced them with path:/command:; this branch already carries the target:/pattern: replacements). Removing the duplicates resolves the validator's "duplicate checkpoint IDs" failure. Verified locally: - `python3 -c "yaml.safe_load(...)"` + dup-id check from netresearch/skill-repo-skill validate.yml: 13 mechanical + 3 llm checkpoints, no duplicates. - `yamllint` (with the validator's inline config) clean. - Manual jq/test runs cover bundle-only, sig+pem, lone-sig, lone-pem, none, annotated-only tags, lightweight tags, and empty repo. Signed-off-by: Sebastian Mendel <github@sebastianmendel.de>
CybotTM
force-pushed
the
feature/standardize-checkpoint-schema
branch
from
28a5393 to
88972c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
path:/content:/command:fields totarget:/pattern:. The runner expects target/pattern (perreferences/checkpoints-schema.md) — the old fields meant every GR-* check returned "Target file not found"; after this change all GR checkpoints actually run..bundle(Sigstore) OR.sig+.pem— the format actually produced bynetresearch/skill-repo-skillrelease workflow.Test plan
bash /home/sme/p/automated-assessment-skill/main/skills/automated-assessment/scripts/run-checkpoints.sh --skill github-release— GR-* checkpoints now produce real PASS/FAIL results instead of "Target file not found".sig+.pemartifacts; PASS on a Sigstore.bundleSigned-off-bypresent