feat(copilot): add evidence-based Copilot customizations#690
Open
YunchuWang wants to merge 15 commits intomainfrom
Open
feat(copilot): add evidence-based Copilot customizations#690YunchuWang wants to merge 15 commits intomainfrom
YunchuWang wants to merge 15 commits intomainfrom
Conversation
Generated by copilot-customizer skill after deep repository inspection.
Each file is justified by direct evidence from the codebase.
Changes:
- .github/copilot-instructions.md: Added two critical missing sections:
(1) Replay Determinism invariant — the most important architectural
constraint in this repo, absent from existing instructions.
Evidence: replay model described in issue-fixer.agent.md,
WARNING/IMPORTANT comments throughout src/.
(2) gRPC/Protobuf layer rules — proto field number permanence,
refresh-protos.ps1 workflow.
Evidence: WARNING comment in JsonDataConverter.cs, refresh-protos.ps1.
(3) Build and test commands — dotnet build/test with exact flags.
Evidence: validate-build.yml CI workflow.
- .github/copilot/instructions/analyzers-generators.md: Path-specific
rules for Roslyn analyzer and source generator code. Different runtime
model (compile-time only), different test infrastructure, interface
consistency constraint with Abstractions.
Evidence: src/Analyzers/Analyzers.csproj (no runtime deps),
IMPORTANT comment in TaskActivityContext.cs.
- .github/copilot/instructions/grpc-worker.md: Path-specific rules for
gRPC proto changes, Worker dispatch loop concurrency constraints, and
the correct pattern for writing integration tests.
Evidence: IMPORTANT comment in TaskOrchestrationDispatcher.cs,
WorkItemDispatcher.cs concurrency note, GrpcSidecarFixture pattern
in test/Grpc.IntegrationTests/.
- .github/copilot/skills/breaking-change-check.md: Skill for systematic
backward-compatibility verification before changing public APIs.
Evidence: copilot-instructions.md already references breaking change
rules but provides no structured process. PR template has a breaking
change checklist. Commit history shows several breaking-change-related
fixes (c299597, e427bbc, 781bb6e).
- .github/copilot/agents/copilot-customizer.md: Interactive Copilot Chat
agent for future customization runs via @copilot-customizer.
Conflict check: No rules duplicated. No contradiction with existing
copilot-instructions.md or the three existing agents.
Co-Authored-By: Claude <noreply@anthropic.com>
Addressing review feedback on the initial customization commit:
1. copilot-instructions.md:
- Remove '## Build and Test' section (commands not appropriate here;
already covered in validate-build.yml which contributors can read)
- Fix '## gRPC / Protobuf Layer': remove false claim that generated
code lives in src/Grpc/Grpc/ — stubs are build-time generated,
not committed. Update to accurate description.
- Add path-change confirmation rule: before referencing or moving
any file path, verify it exists and confirm with user.
2. .github/copilot/agents/copilot-customizer.md:
- Rewrite in correct chatagent fence format matching the existing
agents in this repo (issue-fixer.agent.md pattern).
- Remove @copilot-customizer invocation syntax (Copilot does not
support @agent-name mentions).
- Remove @workspace references (not a supported mechanism in agents).
- Broaden scope: works in Copilot coding agent context, not just
VS Code Copilot Chat.
- Add explicit path-verification rule before any file path is cited.
3. .github/skills/ (moved from .github/copilot/skills/):
- breaking-change-check: moved to .github/skills/breaking-change-check/SKILL.md
to match the correct GitHub Copilot skill directory format
(skills live in named directories as SKILL.md, not flat .md files).
Co-Authored-By: Claude <noreply@anthropic.com>
1. Two-phase model (Issue #2): Added hard stop between Phase 1 (analyze+plan) and Phase 2 (execute). Agent now presents a plan and explicitly waits for user approval with a visible STOP marker before writing any file. 2. Evidence standard hardened (Issue #3): Defined exactly what counts as valid evidence (file:line+snippet, CI command, explicit code comment, CONTRIBUTING.md section, commit pattern). Listed explicit NOT VALID examples (inference, common practice, "likely"). Rule: no evidence → no rule, no file. 3. Path verification with fallback (Issue #4): Added mandatory path verification gate before writing any instruction. If path cannot be verified: do NOT include it, mark as UNVERIFIED PATH, ask user before proceeding. 4. Assessment format loosened (Issue #5): Fields in the plan output are now: only include if supported by hard evidence; write INSUFFICIENT EVIDENCE — omitted otherwise. Prevents filling-the-template behavior. 5. Priority model added (Issue #6): Explicit ordered decision tree (1→5): repo-wide instructions first, AGENTS.md second, path instructions third, skills fourth, agents last. Default: do less. 6. Skill/agent decision rules made deterministic (Issue #7): Replaced descriptive text with ALL-must-be-true checklists for when to create a skill and when to create an agent. 7. Post-run reflection required (Issue #8): Added mandatory Phase 3 post-run reflection covering: mistakes made, over-engineering avoided, hallucinations caught, agent spec improvement suggestions, stale risk. Required on every run. 8. Untestable instruction gate (Issue #9): Added explicit testability gate — if an instruction cannot be violated in a detectable way, it must not be written. 9. Commit fallback for missing git (Issue #10): Phase 2.4 now outputs a commit plan (files, message, concerns) when git is unavailable or commit fails, instead of silently failing. 10. Prohibited language list: Extended and applied as a hard gate — if prohibited language appears in an instruction, the instruction is deleted, not softened. Co-Authored-By: Claude <noreply@anthropic.com>
Issue #1 (shell cmd over-reliance): Every shell block now has an "Else (no terminal)" fallback that instructs the agent to infer from visible context only and to explicitly label each inference — preventing fake command output. Issue #3 (evidence anti-leak): Added Evidence Integrity section: do not fabricate file paths, line numbers, or snippets. If exact line unknown → cite file only. If snippet unavailable → state "snippet not verified". If nothing concrete → do not write the rule. Issue #4 (rigid structure kills reasoning): Plan template changed from "Output this exact structure" to "Use this structure if applicable. Do NOT fabricate content to fill fields." Fields with no evidence write INSUFFICIENT EVIDENCE. Added TERMINAL ACCESS field so confidence level is explicit. Issue #5 (no cost model): Added Cost Model section: each file adds maintenance cost, cognitive load, and contradiction risk. Rule: only add when value > long-term cost. This is now also a required field in the plan (Cost justification per proposed change). Issue #6 (staleness not a decision condition): Staleness Filter is now a pre-write gate, not just post-run output. If a rule is stale-prone within 3 months → move to skill or omit. Stale risk is also a required field in the plan per proposed change. Issue #7 (conflict arbitration incomplete): Added Conflict Resolution Priority table (1=copilot-instructions.md highest, 5=agents lowest). Each proposed change now includes "Authority level: [1-5]" in the plan. Never duplicate across levels. Issue #8 ("under 100 lines" is arbitrary): Changed to "Prefer under 100 lines. If longer, justify why each section earns its place." Applied same principle to path instructions (<60), skills (<150). Issue #9 (commit fallback already OK, confirmed): No change needed — fallback was already added in prior round. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds evidence-based GitHub Copilot customization artifacts to better align AI suggestions with DurableTask .NET SDK invariants (replay determinism, wire compatibility, generator/analyzer constraints) and to provide reusable playbooks/agents for future work.
Changes:
- Expanded
.github/copilot-instructions.mdwith replay determinism and gRPC/proto compatibility guidance. - Added scoped Copilot instruction files for analyzers/generators and gRPC worker/integration-test areas.
- Added a breaking-change review skill and a “copilot-customizer” chat agent spec for future instruction generation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/copilot-instructions.md | Adds repo-wide guidance on orchestrator determinism and gRPC/proto compatibility. |
| .github/copilot/instructions/analyzers-generators.md | Adds path-scoped rules for Roslyn analyzer/generator projects and their tests. |
| .github/copilot/instructions/grpc-worker.md | Adds path-scoped rules for proto wire compatibility, worker dispatch loop, and integration tests. |
| .github/skills/breaking-change-check/SKILL.md | Adds a conditional, multi-step breaking-change safety checklist/playbook. |
| .github/copilot/agents/copilot-customizer.md | Adds a two-phase agent spec for proposing future Copilot customizations with evidence gating. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The closing fence after the WaitForInstanceCompletionAsync example was dropped during rebase conflict resolution. Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- analyzers-generators.md: narrow applyTo to src/Analyzers/**,src/Generators/** only; test/Generators.Tests intentionally uses System.Reflection and Microsoft.Extensions.DependencyInjection (verified in TestHelpers.cs), so the compile-time-only rules must not scope into test projects - grpc-worker.md: add src/Shared/Grpc/ProtoUtils.cs as a required update point when proto fields change; this file performs most history/action conversions and exists alongside the client-side ProtoUtils.cs (verified with find) - copilot-instructions.md: remove proto3-inaccurate 'required fields' phrasing; orchestrator_service.proto uses proto3 (syntax = proto3) which has no required keyword; reworded to describe actual unsafe patterns (fields the SDK/sidecar treats as required, fields without safe defaults, changing field types) Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment on lines
+1
to
+3
| --- | ||
| applyTo: "src/Analyzers/**,src/Generators/**" | ||
| --- |
There was a problem hiding this comment.
applyTo only scopes this instruction file to src/Analyzers/**,src/Generators/**, but the content includes explicit guidance for test/Analyzers.Tests/ and test/Generators.Tests/ (and the PR description lists those tests as in-scope). If the intent is to have these rules active while editing the test projects too, include the test globs in applyTo; otherwise consider removing/relocating the test-specific sections to avoid a misleading scope.
Comment on lines
+26
to
+28
| - Generator tests use `Microsoft.CodeAnalysis.CSharp.SourceGenerators.Testing` — use `GeneratorDriver` or the test helper wrappers in `test/Generators.Tests/`. | ||
| - Compare generated output with explicit expected source strings (inline in the tests or via shared helpers). When changing generator templates, update all affected expected outputs or helper expectations. | ||
| - Do not emit `#pragma warning disable` in generated output without explicit justification. |
There was a problem hiding this comment.
This file says generator tests should compare output with “stored expected snapshots”, but test/Generators.Tests currently asserts generated output via inline expected source strings in TestState.GeneratedSources (e.g., test/Generators.Tests/Utils/TestHelpers.cs). Updating this wording will avoid steering contributors toward a snapshot workflow that the repo doesn’t use today.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Evidence-based GitHub Copilot customization files generated by deep repository inspection of
microsoft/durabletask-dotnet. Every rule is justified by direct repository evidence (code comments, CI workflows, commit patterns). No boilerplate. No fabricated paths or commands. All new rules conflict-checked against the existingcopilot-instructions.md.What Was Added / Modified
.github/copilot-instructions.md— IMPROVED// IMPORTANTcomments inTaskActivityContext.csandTypeExtensions.cs; existingFuncTaskOrchestratorandTaskOrchestrationContextabstractions. Copilot previously had no awareness of the replay constraint and would suggestDateTime.NoworGuid.NewGuid()inside orchestrators.Grpc.Tools,refresh-protos.ps1workflow. Evidence:src/Grpc/orchestrator_service.proto,.github/workflows/*.yml.src/Grpc/refresh-protos.ps1references specific paths; previous issue with assumed paths.## Build and Testsection — build/test commands belong in CI config, not in always-active Copilot instructions..github/copilot/instructions/analyzers-generators.md— CREATEDsrc/Analyzers/**,src/Generators/**,test/Analyzers.Tests/**,test/Generators.Tests/**src/Abstractions/that source generators implement must be reflected in generator output.// IMPORTANT: implemented by source generatorsinTaskActivityContext.cs;src/Generators/andsrc/Analyzers/directory structure..github/copilot/instructions/grpc-worker.md— CREATEDsrc/Grpc/**,src/Worker/**,src/InProcessTestHost/**,test/Grpc.IntegrationTests/**Grpc.Tools— do not commit them; integration tests useGrpcSidecarFixture/IntegrationTestBase.src/Grpc/orchestrator_service.proto,src/Grpc/refresh-protos.ps1,test/Grpc.IntegrationTests/test infrastructure..github/skills/breaking-change-check/SKILL.md— CREATED// WARNINGcomment inJsonDataConverter.cs; breaking change policy linked in existingcopilot-instructions.md;src/<area>/<area>.csproj<PackageId>pattern for shipped NuGet packages..github/copilot/agents/copilot-customizer.md— CREATEDEvidence Highlights
TaskOrchestrationContextabstractions;FuncTaskOrchestrator;// IMPORTANTinTaskActivityContext.cssrc/Grpc/orchestrator_service.protofield numberingsrc/Grpc/*.csprojreferencesGrpc.Tools; no.csfiles insrc/Grpc/that are stubs// IMPORTANT: implemented by source generatorsinTaskActivityContext.cscopilot-instructions.mdlinksdotnet/runtimebreaking-change-rulesConflict Check
All new rules were checked against the existing
.github/copilot-instructions.md. No duplicates or contradictions introduced. The Replay Determinism and gRPC sections incopilot-instructions.mdextend (not replace) the existing C# guidelines. Path-specific instructions apply only to their scoped directories and do not override global rules.Maintenance Notes
copilot-customizerquarterly or after major architectural changesanalyzers-generators.mdif new source generator projects are addedgrpc-worker.mdif the protobuf transport is replaced or supplementedbreaking-change-checkskill should be reviewed when the NuGet release process changesLimitations
.github/copilot/instructions/) require Copilot for Business or Enterprise tiercopilot-customizeragent requires Copilot Chat with agent support enabledFuture Follow-ups
DateTime.Nowin orchestrator contexts without this guidancerefresh-protos.ps1path reference (if proto tooling changes);Grpc.Toolspackage namesrc/Clients/andsrc/Extensions/— insufficient evidence of rules that must not apply globally;AGENTS.md— no autonomous coding agents actively used in this repo's documented workflowapplyToglobs on the path-specific instruction files; adjust scope patterns if needed🤖 Generated with Claude Code via
copilot-customizerskill