AI & ML
Your team's coding rules aren't in the prompt, they're in the ingest
Cole Halton DEV Community
2 views
Every AI code reviewer claims it respects your team's standards. Very few can tell you what those standards actually are.
The test is mechanical. Ask the tool, or the vendor, one question: where do my rules live? If the honest answer is "we loaded a generic style guide plus whatever your PR description happened to say", then your standards aren't in the model at all. The reviewer is running on vibes and hoping your repo looks conventional enough to pass.
The tools that genuinely track your rules share a shape: the standards are an input, not a hope. They read your rule files, your linter config, your past review comments, sometimes your docs. The review is judged against that artifact, which means when a comment fires you can ask "which rule?" and get a config line back, not a paragraph of model reasoning.
That last part is the thing worth testing for. If a reviewer can't point to the specific rule it applied, it is not following your standards. It is approximating what it assumes standards look like. Those two feel identical for the first six months, then diverge exactly when you've stopped proofreading its output.
A short checklist when you trial one:
Does it ingest a rules file, or only the PR context?
Can it point to the exact rule that triggered a comment?
Does it adapt to your historical review style, or reset every run?
Is a "rule" something you can open, read, and edit in the UI?
If the answer to all four is no, you bought a very chatty spellchecker that happens to be trained on GitHub. The eval that actually decides it: can your own reviewer reproduce one of your team's real past review decisions, given only your real rules file? Run that before you hand it a production PR.
Read original: https://dev.to/cole_halton_42f71d71b809b/your-teams-coding-rules-arent-in-the-prompt-theyre-in-the-ingest-3jp0
← Previous
The GNU strip backdoor is the case AI code review can't see
Next →
How to Send Email from Cloudflare Workers
Related
A Small, Checkable Test for AI Memory Systems
AI & ML
3
DEV Community
The GNU strip backdoor is the case AI code review can't see
AI & ML
1
DEV Community
I Want More Coding Agents to Work Like This
AI & ML
1
DEV Community
Gemini Often Won't Search the Web — and Won't Tell You It Didn't
AI & ML
2
Dev.to (EN Zone)
Comments0
No comments yet — be the first