Move beyond comment counts with a measurement system for accepted and dismissed findings, precision, coverage samples, severity, maintainer effort, latency, reliability, and cost.
AI code review quality cannot be summarized by the number of comments posted. A reviewer that writes more will appear active while consuming more maintainer time. It also cannot be summarized by acceptance rate alone: teams may accept harmless cleanup to close a thread, and a precise reviewer may miss important defects entirely. Measurement needs to represent both the usefulness of what was said and the risk of what was missed.
Start with a finding-level event model. Every finding should have a stable identifier, repository, pull request and head SHA, reviewer role, model and prompt version, severity, confidence, file, line context, creation time, and resolution. Keep the originally generated content even if the posted display is edited or aggregated. Without version and provenance fields, a configuration change can make a trend look like improvement or regression without revealing why.
Accepted should mean the finding caused a code, test, or documentation change, or caused a deliberate risk decision. Dismissed should not be one undifferentiated bucket. Ask for lightweight reasons: incorrect, already handled, out of scope, duplicate, not actionable, accepted risk, or superseded by later code. Resolution can be inferred from comment threads only imperfectly, so offer maintainers a quick explicit control when possible.
Routine production data reveals false positives but not defects the reviewer never mentioned. To estimate coverage, periodically sample merged pull requests and have knowledgeable humans label material issues visible in the diff and relevant context. Historical defect-fix pull requests can also test whether the reviewer identifies a known failure, but avoid leaking the later fix or old review comments into the input. This process is expensive, so use a small, representative set and document its limitations.
Segment results before interpreting them. A security reviewer on authentication code is a different task from an architecture reviewer on a documentation change. Compare repository, language, change type, diff size, role, model, prompt version, and confidence band. Aggregate numbers can hide a high-value niche reviewer behind a larger volume of irrelevant runs or make a model look strong because it received easier pull requests.
Ask maintainers periodically whether the review was worth reading and whether it changed the time needed to reach a merge decision. Instrument carefully: time from comment to resolution is affected by work schedules and does not equal review effort. Qualitative thread audits are indispensable. Read examples of accepted, rejected, and ignored findings to learn whether the categories match reality and whether comments explain enough evidence for fast decisions.
Version configuration and change one major variable at a time when practical. Raising the confidence floor should reduce volume; inspect whether noise falls without eliminating high-value findings. Switching a model may change schema validity and confidence calibration as well as reasoning quality. A dashboard is useful only if it supports decisions: narrow a role, exclude a path, change a model, adjust a cap, or stop a reviewer that does not earn its cost. Measurement should make AI review accountable to the team’s attention.
Avoid publishing one quality score without its denominator and uncertainty. A repository with twelve resolved findings cannot support the same conclusion as one with hundreds, and both may be biased by who chose to resolve comments. Show raw counts beside rates, document the sampling method, and annotate configuration changes. Honest measurement is slower, but it prevents a precise-looking chart from becoming permission to deploy a reviewer beyond the evidence.
PR Quorum turns specialist reviewer output into one clean GitHub review, with noise controls and predictable usage caps.