Most AI code reviewers work the same way: point an LLM at a diff, ask it to flag problems, ship the output. It looks impressive in a demo and falls apart the moment you rely on it — because the model's confidence doesn't track its correctness. It'll miss an obvious governor-limit bug in a loop and then confidently invent a problem that isn't there, with the same tone either way.
When I built ApexDebugger, a reviewer for Salesforce Apex and Lightning Web Components, I didn't want a tool that "usually" catches the bugs that matter. So the design starts from one rule:
Bounded-and-decidable → code. Unbounded-judgment → LLM.
If a check has a yes/no answer that a human could verify by inspection —
SOQL or DML sitting inside a loop, a hardcoded record ID, a missing
WITH SECURITY_ENFORCED or sharing declaration — it doesn't belong
in a model call. It belongs in a regex or an AST check that returns the same
answer every time, for free, with 100% recall on what it's built to catch.
That's the deterministic layer.
Everything else — is this method doing too much, is this exception handling actually risky, is this the third near-duplicate of the same helper — doesn't have a bright-line answer. That's a judgment call, the same kind a senior reviewer makes by reading the code and thinking about intent. That's where the LLM reasoning layer earns its keep, running through LangGraph with structured output against a shared rule taxonomy.
The objective checks live in the certain layer. The judgment calls go to the layer built to make judgment calls. Neither one is asked to do the other's job.
Making the LLM layer honest
A hybrid system is only as good as its weakest layer, and LLMs hallucinate. So most of the actual engineering here isn't the review logic — it's constraining the model layer until its mistakes become measurable and rare:
- Blind independent review. The LLM never sees the deterministic layer's findings before it reviews. No anchoring — it's a genuine second opinion, not a rubber stamp on rules already found.
- Consensus voting. Every LLM finding runs three independent times; only findings that show up in at least two of three survive. Random hallucinations flicker between runs. Real findings hold steady — this collapsed run-to-run variance to close to zero on most of the golden set.
- Regex authority. For any rule type the deterministic layer owns, its verdict is final — an LLM claim on that same rule gets dropped. This kills the hallucinations that voting can't: the ones that are wrong but consistently wrong.
-
Controlled vocabulary. Both layers speak through the same
RuleIdenum, so every finding is dedupable and scoreable instead of being two systems producing free-text that has to be reconciled by hand.
The eval is the actual product
None of that is worth anything without a way to check it's true. Every rule type is scored against a hand-labeled golden set — review judgment frozen into data instead of left as a vibe. There are two gates:
- A deterministic gate that runs the regex layer against the golden labels — free, reproducible, runs in CI on every PR. It currently sits at 17 of 17 golden cases at 1.00 precision and recall.
- A probabilistic eval that runs the full pipeline multiple times per case and reports mean F1 and spread. Spread turns out to be a useful diagnostic on its own: low spread with a low score means a deterministic bug — fix the code. High spread means LLM noise — fix the prompt. Same failing score, different fix, and the spread number is what tells you which one you're looking at.
I also A/B tested grounding the LLM in general Salesforce best-practice docs versus not. Zero measurable lift — the model already knows generic best practice. So grounding effort now goes where the model actually has a gap: project- and org-specific conventions it has no way to know on its own.
What this costs, measured
Because only the LLM layer costs money, the whole thing is cheap to run on every push: roughly $0.0035 per pull request, measured rather than estimated. A two-file PR runs about ten model calls — Apex reasoning with three-way voting, LWC reasoning with three-way voting, cross-file reasoning with three-way voting, plus one rollup summary — on the order of 16.8K tokens. The deterministic layer and the CI gate cost nothing at all.
Why this matters beyond one tool
I spent eight years as a Salesforce architect doing the review work this tool now automates — reading Apex for governor-limit risk, checking sharing and field-level security, chasing down the same class of bug across a hundred pull requests. Encoding that judgment into something measurable, instead of something that just sounds right, is the actual engineering problem. The AI part is the easy half.