Ask any frontier model to implement binary search, and the answer will look impressive. Ask it to change a shared utility used by fourteen services without breaking callers, and the differences become much less polite. That is the problem with most “GPT-6 vs. Claude 5.1 vs. Gemini” comparisons. They test the wrong surface area. A model can look brilliant on a self-contained algorithm prompt while still being expensive in a real repository: it may invent imports, ignore constraints, swallow errors, produce sprawling diffs, or write tests that only cover the happy path. The useful question is not: Which model writes the prettiest function? The useful question is: Which model fails in the least damaging way when the task is ambiguous, constrained, and embedded in an existing system? That is what a code test should measure. TL;DR Do not compare frontier coding models using only toy prompts. Run a battery of code tests: ambiguity handling, bug repair, constrained refactoring, error handling, multi-file changes, security review, test generation, and strict-type migration. Score models on constraint adherence, diff size, hallucinated APIs, edge-case awareness, and failure behavior. Vendor benchmarks are useful signals, but they are not production truth. The best model for your team depends on whether your bottleneck is prototyping speed, correctness, repository understanding, review cost, or operational safety. 📋 Table of Contents The Wrong Way to Compare Frontier Coding Models The Test Harness That Makes the Comparison Useful 1. The Ambiguous Rate Limiter 2. The One-Line Bug That Exposes Patch Discipline 3. The “Do Not Change the Public API” Refactor 4. The Retry Test Where Most Models Over-Retry 5. The Multi-File Change That Reveals Repo Awareness 6. The Security Prompt That Separates Confidence From Correctness 7. The Test-Generation Test That Reveals Edge-Case Thinking 8. The Strict-Type Migration Test Comparison Table: What Actually Matters A Practical Way to Choose The Wrong Way to Compare Frontier Coding Models A typical model comparison goes like this: Ask each model to solve a LeetCode-style problem. Ask each model to write a small REST endpoint. Ask each model to explain a concept. Declare a winner based on which output reads best. This is weak evidence for production use. A good-looking answer can still be wrong in the ways that cost engineering time: It introduces a dependency the project does not use. It changes a public interface that other modules rely on. It assumes a database behavior that was never specified. It retries requests that should not be retried. It writes tests that assert the implementation instead of the behavior. It produces a diff so large that review becomes a separate project. When comparing GPT-6, Claude 5.1, and Gemini, the more valuable signal is behavior under constraint. A model that asks a clarifying question may be more useful than one that confidently writes fifty lines of code. A model that makes a minimal fix may be more valuable than one that refactors half the file. A model that refuses to guess a database schema may save you from a subtle production incident. That is especially true in 2026, where coding assistants are rarely used as isolated chat windows. They are embedded in IDEs, agents, pull-request reviewers, terminal tools, CI pipelines, and internal platforms. The model is not just generating code; it is participating in a workflow. So the code test needs to resemble that workflow. The Test Harness That Makes the Comparison Useful Before looking at individual prompts, create a scoring structure. Otherwise, the comparison collapses into vibes. A simple evaluation record can look like this: from dataclasses import dataclass, field @dataclass class ModelTrial: model: str task: str passed_tests: bool made_unnecessary_changes: bool hallucinated_api: bool asked_clarifying_questions: bool stated_assumptions: bool preserved_public_api: bool introduced_security_issue: bool test_quality: int notes: list[str] = field(default_factory=list) The exact fields matter less than the discipline of recording them. For each task, capture: The exact prompt. The exact model identifier or pinned version. The temperature or sampling settings. Whether the model asked questions or assumed answers. Whether the code ran without modification. Whether the model changed more than requested. Whether the model invented functions, fields, libraries, or environment variables. Whether the output would survive code review. ⚠️ Gotcha: Do not compare “latest” aliases across vendors without recording the underlying model version. A leaderboard built on mutable aliases can become meaningless after a provider update. A useful comparison is not “Claude 5.1 is better than Gemini” in the abstract. It is: Claude 5.1 may be preferable when the task requires careful constraint adherence. Gemini may be preferable when the task benefits from broad repository context or ecosystem integration. GPT-6 may be preferable when the task requires rapid generation of practical implementation code. But those preferences should come from your tasks, not from a generic prompt gallery. The following tests are designed to surface those differences. 1. The Ambiguous Rate Limiter Scenario: You ask a model to implement a rate limiter for an API. The prompt is intentionally incomplete. It does not specify whether the rate limiter should be fixed window, sliding window, token bucket, in-memory, distributed, user-based, IP-based, or durable across restarts. Prompt: Implement a rate limiter for a Python API. It should allow 100 requests per minute per user. Do not assume a framework. This looks simple. It is not. A weak model will immediately choose an architecture and hide the assumptions inside the implementation. A stronger model will either ask questions or explicitly state the assumptions it is making. What to score: Does the model ask about distributed versus single-process use? Does it state whether it is using a fixed window or sliding window? Does it choose an in-memory store when no persistence requirement exists? Does it avoid dragging in Redis, Kafka, or a database unnecessarily? Does it expose a simple interface that can be replaced later? A reasonable minimal implementation might look like this: from time import monotonic class FixedWindowRateLimiter: def __init__(self, limit: int, window_seconds: int = 60): self.limit = limit self.window_seconds = window_seconds self.counters: dict[str, tuple[int, int]] = {} def allow(self, user_id: str) -> bool: now = monotonic() window_id = int(now // self.window_seconds) current_window, count = self.counters.get(user_id, (window_id, 0)) if current_window != window_id: current_window = window_id count = 0 if count >= self.limit: self.counters[user_id] = (current_window, count) return False self.counters[user_id] = (current_window, count + 1) return True This implementation is not the “best” rate limiter. It is a controlled answer to an ambiguous prompt. Why this works: The code is small enough to inspect, but it forces the model to make decisions about time, storage, and identity. Those decisions are where real engineering judgment appears. What to watch for: If the model immediately introduces Redis without mentioning trade-offs, it may be overengineering. If the model uses datetime.now() without explaining timezone or clock behavior, it may be underthinking concurrency and restarts. If the model asks whether the limiter must be distributed, it is showing production awareness. 💡 Practical note: For this test, a model that says “I assume a single-process in-memory fixed window” is often more useful than one that silently builds a complicated distributed solution. 2. The One-Line Bug That Exposes Patch Discipline Scenario: You give the model a small function with a realistic bug. The task is not to redesign the module. The task is to fix the bug with the smallest safe change. Prompt: This pagination function is supposed to use 1-based page numbers. It currently returns the wrong page. Fix the bug with the smallest change possible. Do not refactor unrelated code. Buggy code: def paginate(items, page: int, per_page: int): start = page * per_page end = start + per_page return items[start:end] If page is 1-based, requesting page 1 with per_page=10 starts at index 10 instead of index 0. A disciplined fix: def paginate(items, page: int, per_page: int): if page < 1: raise ValueError("page must be greater than or equal to 1") if per_page < 1: raise ValueError("per_page must be greater than or equal to 1") start = (page - 1) * per_page end = start + per_page return items[start:end] Why this matters: Many models are eager to “improve” code even when the instruction is to make a minimal fix. They may rewrite the function, add generics, introduce a paginator class, or convert the code to use a library. That can be harmful in a real repository because the diff becomes harder to review and the change may affect callers in unexpected ways. What to score: Did the model fix the actual bug? Did it preserve the function signature? Did it add validation that was reasonable, or did it expand scope too far? Did it explain why the original code was wrong? Did it avoid changing unrelated behavior? Failure mode to watch for: A model may return something like this: from typing import Iterable, TypeVar from dataclasses import dataclass T = TypeVar("T") @dataclass class PageResult: items: list page: int per_page: int total: int That may be a nice abstraction, but it is not the requested fix. This test reveals whether the model can operate like a careful engineer or whether it behaves like an enthusiastic refactoring engine. 3. The “Do Not Change the Public API” Refactor Scenario: You need an internal cleanup, but the public interface must remain stable. This is common in library code, SDKs, internal platform utilities, and shared modules. Prompt: Refactor the internals of this module without changing any exported function signatures. Do not change the public types. Do not change the behavior visible to callers. Example TypeScript module: export interface User { id: string; name: string; email: string; archived: boolean; } export function getUserLabel(user: User): string { return formatLabel(user); } function formatLabel(user: User): string { if (user.archived) { return `${user.name} (archived)`; } return user.name; } Now ask the model to add an internal helper for redacting email addresses, or to move formatting logic into a separate internal function. A good model will keep this stable: export function getUserLabel(user: User): string { return formatLabel(user); } It may add internal helpers: function formatLabel(user: User): string { return user.archived ? withArchivedSuffix(user.name) : user.name; } function withArchivedSuffix(name: string): string { return `${name} (archived)`; } A bad model will change the exported function: export function getUserLabel(user: User, includeArchived: boolean): string { // ... } That breaks callers. Why this works: Public API stability is one of the most important constraints in real code. Models that do not respect it create hidden costs. The code may compile locally, but downstream services, notebooks, scripts, or frontend modules may fail. What to score: Did the model preserve exported names? Did it preserve argument order and types? Did it avoid changing return shapes? Did it keep tests passing? Did it separate internal helpers from public surface area? 🔍 Why this matters: In a monorepo, a model that changes one public type can generate a review burden far larger than the original task. This is one of the best tests for comparing GPT-6, Claude 5.1, and Gemini in practical engineering work. A model that understands “do not change the public API” is more trustworthy than one that only understands “make the code nicer.” 4. The Retry Test Where Most Models Over-Retry Scenario: You ask the model to implement an HTTP fetch helper with retries. This seems routine, but retry logic is easy to get wrong. Prompt: Write a Python function that fetches a URL using requests. Retry on temporary network failures and 5xx responses. Do not retry on 4xx responses except 429. Use exponential backoff with jitter. Do not retry forever. A reasonable implementation: import random import time import requests RETRYABLE_STATUS = {429, 500, 502, 503, 504} def fetch_with_retry( url: str, *, attempts: int = 5, base_delay: float = 0.5, timeout: float = 5.0, ): for attempt in range(attempts): delay = base_delay * (2 ** attempt) + random.uniform(0, 0.1) try: response = requests.get(url, timeout=timeout) if response.status_code in RETRYABLE_STATUS: if attempt == attempts - 1: response.raise_for_status() time.sleep(delay) continue response.raise_for_status() return response except (requests.ConnectionError, requests.Timeout): if attempt == attempts - 1: raise time.sleep(delay) raise RuntimeError("unreachable") What to score: Does it distinguish temporary failures from permanent failures? Does it avoid retrying 400, 401, 403, 404, and 422? Does it retry 429 carefully? Does it add jitter? Does it cap attempts? Does it use timeouts? Does it avoid retrying non-idempotent mutations blindly? Why this matters: A model that retries everything can turn a small downstream outage into a thundering herd. A model that retries POST requests without understanding idempotency can create duplicate payments, duplicate messages, or duplicate resource creation. Common model failure modes: Retrying all exceptions. Retrying all HTTP errors. Sleeping without jitter. Using unbounded recursion. Ignoring timeouts. Assuming requests will retry automatically. Inventing a nonexistent retry helper from a library that was not imported. This test is especially useful because it separates surface-level competence from operational maturity. The code can look correct while being dangerous. 🚨 Production warning: If the model does not mention idempotency when asked to retry POST, PUT, or PATCH requests, treat its networking advice with caution. 5. The Multi-File Change That Reveals Repo Awareness Scenario: The task requires changing one module while respecting the boundaries of another. This is where many coding models start to hallucinate. Give the model a small set of files instead of a single snippet. Example file layout: app/ orders/ service.py repository.py billing/ events.py Starting code: # app/orders/service.py class OrderService: def __init__(self, repo, events): self.repo = repo self.events = events def cancel_order(self, order_id: str, reason: str | None = None) -> None: self.repo.mark_cancelled(order_id) Prompt: Modify OrderService.cancel_order so that it publishes an OrderCancelled event. Do not import billing code directly. Do not change the existing public method signature. Prefer an event object over adding many arguments. A strong response may introduce an event type: from dataclasses import dataclass @dataclass(frozen=True) class OrderCancelled: order_id: str reason: str | None Then publish it: class OrderService: def __init__(self, repo, events): self.repo = repo self.events = events def cancel_order(self, order_id: str, reason: str | None = None) -> None: self.repo.mark_cancelled(order_id) self.events.publish(OrderCancelled(order_id=order_id, reason=reason)) What to score: Did the model respect module boundaries? Did it import something it was told not to import? Did it invent fields on the event object? Did it change the repository interface unnecessarily? Did it preserve the public method signature? Did it understand that events should flow through an event publisher or outbox? Did it avoid creating a circular dependency? Why this works: Real code is not one function. It is a network of files, contracts, ownership boundaries, and implicit rules. A model that only sees the current function can produce code that is locally correct but architecturally damaging. This is one of the most important tests for Gemini-style large-context workflows, Claude-style constrained reasoning, and GPT-style practical code generation. The difference is not just whether the code compiles. The difference is whether the model understands the repository as a system. 6. The Security Prompt That Separates Confidence From Correctness Scenario: You give the model a piece of code that appears to work but contains a common vulnerability. Then you ask it to review or fix the code. A good example is path traversal. Unsafe code: import os def read_user_file(base_dir: str, filename: str) -> str: path = os.path.join(base_dir, filename) with open(path, "r", encoding="utf-8") as file: return file.read() At first glance, this looks harmless. But if filename is something like ../../etc/passwd, the function may read files outside the intended directory. A safer implementation: from pathlib import Path def read_user_file(base_dir: str, filename: str) -> str: base = Path(base_dir).resolve() target = (base / filename).resolve() if not target.is_relative_to(base): raise ValueError("invalid file path") return target.read_text(encoding="utf-8") What to score: Does the model identify the vulnerability? Does it explain why the original code is unsafe? Does it use path canonicalization or resolution? Does it reject path traversal instead of merely sanitizing one pattern? Does it avoid inventing nonexistent security libraries? Does it preserve the intended function behavior? Another useful security test is SQL injection: query = f"SELECT * FROM users WHERE email = '{email}'" A correct fix uses parameterized queries: cursor.execute("SELECT * FROM users WHERE email = ?", (email,)) Why this matters: Security mistakes are often confident. A model can produce a polished explanation while leaving the vulnerability intact. This test reveals whether the model can reason about adversarial input or merely pattern-match common safe-looking code. Failure modes: Replacing os.path.join with string concatenation. Blocking only ../ instead of resolving the final path. Assuming the filename comes from a trusted source. Adding a regex that can be bypassed. Suggesting a dependency without explaining whether it solves the issue. This test is not about making the model paranoid. It is about checking whether it can produce code that survives contact with untrusted input. 7. The Test-Generation Test That Reveals Edge-Case Thinking Scenario: You ask the model to generate tests for a small utility function. This is deceptively revealing. Many models can write tests for the obvious case. Fewer models can identify the edge cases that actually break production code. Function under test: def parse_duration(value: str) -> int: units = { "s": 1, "m": 60, "h": 3600, } value = value.strip() if not value: raise ValueError("duration cannot be empty") unit = value[-1] if unit not in units: raise ValueError("unsupported duration unit") number = int(value[:-1]) if number < 0: raise ValueError("duration cannot be negative") return number * units[unit] Prompt: Write pytest tests for parse_duration. Cover valid inputs, invalid inputs, boundary cases, and error cases. Do not only test the happy path. A decent test suite: import pytest from durations import parse_duration def test_parses_seconds(): assert parse_duration("5s") == 5 def test_parses_minutes(): assert parse_duration("2m") == 120 def test_parses_hours(): assert parse_duration("1h") == 3600 def test_rejects_empty_string(): with pytest.raises(ValueError): parse_duration("") def test_rejects_unknown_unit(): with pytest.raises(ValueError): parse_duration("5x") def test_rejects_negative_duration(): with pytest.raises(ValueError): parse_duration("-5s") def test_rejects_missing_number(): with pytest.raises(ValueError): parse_duration("s") def test_handles_whitespace(): assert parse_duration(" 10s ") == 10 What to score: Does the model test valid units? Does it test empty input? Does it test unknown units? Does it test negative values? Does it test whitespace? Does it test missing numeric parts? Does it test very large values if relevant? Does it avoid asserting implementation details? Why this works: Test generation is not just a coding task. It is a reasoning task about failure modes. A model that only writes happy-path tests is not saving much time. A model that identifies invalid inputs, malformed inputs, and boundary conditions is genuinely useful. This is also a good test for whether the model understands the difference between testing behavior and testing internals. If the model asserts that a dictionary contains a certain key inside the function, that is usually a bad sign. 8. The Strict-Type Migration Test Scenario: You have a small JavaScript utility and you want to migrate it to TypeScript with strict typing. This tests whether the model can improve correctness without changing runtime behavior. Starting JavaScript: export async function getJSON(url) { const response = await fetch(url); return response.json(); } Prompt: Convert this JavaScript function to TypeScript. Assume strict mode. Preserve the runtime behavior. Do not add a framework. Make the return type generic if appropriate. Handle non-OK responses. A reasonable TypeScript version: export async function getJSON<T>(url: string): Promise<T> { const response = await fetch(url); if (!response.ok) { throw new Error(`Request failed with status ${response.status}`); } return (await response.json()) as T; } What to score: Does it add a generic return type? Does it type the URL parameter? Does it preserve the original behavior where possible? Does it handle non-OK responses? Does it avoid inventing a browser-only or Node-only API? Does it use unknown instead of any where appropriate? Does it explain the limitations of as T? A more cautious version may return unknown: export async function getJSON(url: string): Promise<unknown> { const response = await fetch(url); if (!response.ok) { throw new Error(`Request failed with status ${response.status}`); } return response.json(); } That version is less ergonomic but more honest, because the runtime cannot magically validate the JSON shape. Why this matters: Migration work is common. Teams rarely adopt a new model by asking it to create a greenfield project from scratch. They ask it to upgrade JavaScript to TypeScript, convert class components to hooks, move from CommonJS to ESM, update test frameworks, or migrate from one ORM version to another. This test reveals whether the model can improve static safety while respecting runtime behavior. It also reveals whether the model understands that type assertions are not runtime validation. Comparison Table: What Actually Matters A useful comparison between GPT-6, Claude 5.1, and Gemini should not be a single score. It should be a matrix of behaviors. Test What It Reveals Good Signal Bad Signal Ambiguous rate limiter Assumption handling States assumptions or asks questions Silently chooses complex architecture One-line bug fix Patch discipline Minimal safe diff Rewrites unrelated code Public API refactor Constraint adherence Preserves exported signatures Changes public types or arguments Retry logic Operational safety Retries only safe failures Retries all errors forever Multi-file change Repository awareness Respects module boundaries Hallucinates imports or creates cycles Security review Adversarial reasoning Fixes root cause Applies superficial sanitization Test generation Edge-case thinking Covers invalid and boundary cases Only tests happy path Strict-type migration Type-safety judgment Improves correctness without changing behavior Adds unsafe assertions casually You can also compare models across engineering dimensions: Dimension Why It Matters What to Measure Correctness Code must work Tests passed, runtime errors, edge cases Constraint adherence Real work has rules Prompt instructions followed Diff size Review cost Lines changed, unrelated edits Hallucination rate Trust Fake APIs, invented fields, nonexistent imports Clarification behavior Ambiguity handling Questions asked, assumptions stated Security posture Risk reduction Vulnerabilities found and fixed Test quality Maintainability Edge cases, assertions, isolation Latency and cost Workflow fit Time per task, tokens used, retry count The last row is important. A model that produces slightly better code but takes three times longer and uses significantly more tokens may not be the right default for an IDE assistant. A model that is slightly weaker but faster and more predictable may be more valuable for high-volume internal tooling. A Practical Way to Choose Do not choose a coding model based on a single task. Choose based on the shape of your work. If your team mostly builds greenfield prototypes Prioritize: Speed of generating plausible scaffolding. Ability to produce readable examples. Good default structure. Low friction with common frameworks. Useful explanations. Tests to weight heavily: Ambiguous rate limiter. Strict-type migration. Basic endpoint generation. Here, a model that is slightly more creative may be worth a higher review cost. If your team works in a large existing repository Prioritize: Respect for file boundaries. Minimal diffs. Accurate imports. Avoidance of hallucinated symbols. Ability to follow “do not change” instructions. Tests to weight heavily: Public API refactor. Multi-file change. One-line bug fix. In this environment, a model that changes too much is not just annoying. It becomes a code-review tax. If your team builds payment, security, or infrastructure code Prioritize: Edge-case awareness. Safe error handling. Security reasoning. Conservative assumptions. Explicit failure behavior. Tests to weight heavily: Retry test. Security review. Test generation. Here, confidence is less important than correctness. A model that says “this requires clarification” can be more valuable than one that guesses. If your team uses agents inside CI or review pipelines Prioritize: Structured output. Determinism. Low hallucination rate. Clear explanations. Stable behavior across runs. Tests to weight heavily: Bug fix with minimal diff. Security review. Test generation. Agent workflows amplify both strengths and weaknesses. If a model produces a bad fix once in a while, a human may catch it. If an agent applies that bad fix automatically across dozens of pull requests, the damage scales. A reasonable scoring rubric For your own repository, you could use something like this: Category Weight Correctness against tests 35% Constraint adherence 25% Security and error handling 15% Diff size and review cost 10% Clarification and assumption quality 10% Latency and cost 5% Adjust the weights based on your actual bottleneck. If review time is your biggest problem, increase diff size and constraint adherence. If production incidents are your biggest problem, increase security and error handling. If developer velocity is the problem, increase scaffolding and migration performance. The point is not to crown a permanent winner among GPT-6, Claude 5.1, and Gemini. The point is to build a comparison that reflects the work you actually do. A frontier model can look magical in a demo and still be the wrong choice for your repository. The code test is not about finding the smartest answer. It is about finding the model whose failures you can survive, review, and correct with the least cost.