Build a useful feedback loop from finding resolutions without confusing accepted comments with truth, dismissals with failure, or easily gamed rates with engineering quality.
Acceptance and dismissal metrics are among the most accessible signals for an AI code reviewer because they arise inside the normal pull-request workflow. They are also easy to misuse. An accepted comment is not automatically correct or important; an author may make a harmless change to end a thread. A dismissed comment is not automatically worthless; it may expose undocumented context or an explicitly accepted risk. The metric needs a resolution vocabulary and periodic human audit.
Define the unit as a finding, not a review. One review may contain several inline findings and a summary. Give each finding an immutable identifier and store its reviewer role, model, prompt and configuration versions, severity, confidence, evidence location, and head SHA. When aggregation merges duplicates, preserve the source records but count the posted decision once. Otherwise a multi-agent system can inflate both volume and apparent agreement.
Accepted should mean that the finding caused a change or explicit risk decision. Break dismissal into reasons that support action: incorrect, already handled, duplicate, out of scope, not actionable, unclear, accepted risk, or stale after a new commit. Include unresolved and ignored findings rather than dropping them from the denominator. A high apparent acceptance rate built from only the comments people bothered to resolve is selection bias.
Sample comment threads and changed code. Did an accepted finding actually cause a meaningful fix, or only wording cleanup? Was an incorrect dismissal supported by repository context? Were unresolved findings simply abandoned after merge? A small qualitative audit catches incentive problems and UI ambiguity that dashboards cannot. It also supplies examples for prompt evaluation without inventing synthetic customer evidence.
Segment before comparing. Repository risk, pull-request size, change type, programming language, reviewer role, model, prompt version, and confidence band all influence outcomes. Do not rank individual developers by how often they accept AI comments; that invites compliance behavior and makes disagreement costly. Use the data to tune the system, not evaluate the people whose judgment the system depends on.
Resolution metrics observe only what the reviewer said. A silent reviewer can achieve perfect precision while missing every defect. Periodically label a small sample of pull requests or known historical defects and record material issues the reviewer should have surfaced. This gives an imperfect coverage estimate. Keep the sample separate from production acceptance metrics and document how examples were selected.
Repeated out-of-scope architecture findings suggest a narrower role or path policy. Duplicates suggest aggregation work. Already-handled findings suggest missing retrieved context. High-confidence incorrect claims suggest recalibration or a model change. After updating configuration, compare the new version on the same evaluation set and monitor live resolutions. Acceptance and dismissal data is valuable when it leads to fewer unjustified interruptions, not when it becomes a score everyone learns to optimize.
Set a review cadence appropriate to volume. A small team may inspect examples monthly; a large rollout may automate weekly segment reports and perform a deeper quarterly audit. Trigger an earlier review after model, prompt, pricing, or scope changes. Retain enough history to compare versions while honoring repository data policies. Metrics become governance when somebody owns the cadence and the resulting configuration decisions.
PR Quorum turns specialist reviewer output into one clean GitHub review, with noise controls and predictable usage caps.