I started building a website audit tool with a simple plan: crawl a site, run a list of checks, calculate a score, show the fixes. That plan lasted until the first slow website. A page timed out. The report marked the site as failed. I ran the same audit again and got a different result. The site had not changed. The network had. That was the point where the scoring formula stopped being the hardest part. The harder problem was deciding whether a result described the website or described my runtime. This post covers the design of the audit engine behind AdSenseReady. It runs 41 checks across security, content, structured data, performance, mobile UX, and policy signals. The product helps publishers prepare sites for AdSense review, but the engineering problem applies to any tool that turns a crawl into a quality report. A check needs more than pass or fail The first version of the result type looked like this: type CheckStatus = 'pass' | 'warn' | 'fail'; interface CheckResult { checkId: string; status: CheckStatus; message: string; } It looked tidy. It also lied. A check can fail because a site has a bad security header. It can fail because the request to the site timed out. Those outcomes need different advice. The first one belongs in the report as a fix. The second one belongs in the report as a limitation of the audit. The current result model carries two extra states and a verification field: type CheckStatus = 'pass' | 'warn' | 'fail' | 'skip' | 'info'; type VerificationState = | 'verified' | 'manual-confirmation' | 'advisory-only' | 'environment-limited' | 'not-applicable'; interface CheckResult { checkId: string; status: CheckStatus; verificationState?: VerificationState; message: string; details?: Record<string, unknown>; } skip means the engine could not evaluate the check. info means the check ran and produced a useful note without creating a problem. environment-limited explains why a result should not influence readiness scoring. That extra vocabulary fixed a product problem too. Users can see the difference between “your page has no canonical URL” and “the audit could not fetch the page.” A score without that distinction asks users to trust a number they cannot inspect. The crawler builds an audit budget The crawler does not fetch every URL it can discover. Each plan gets a page budget and a crawl configuration. A crawl can include: the homepage core pages such as /privacy, /about, /contact, /terms, and /faq URLs from /sitemap.xml internal links found on the homepage The homepage counts toward the page limit. That detail caused a small but embarrassing bug. A plan capped at 100 pages fetched the homepage plus 100 discovered pages. The report said 101 pages. Users noticed before I did. The fix was to reserve the homepage slot before building the final queue: const seen = new Set(pages.map((page) => normalizeUrl(page.url))); const finalUrls = []; const remaining = config.maxPages - pages.length; for (const item of queue) { if (finalUrls.length >= remaining) break; const key = normalizeUrl(item.url); if (seen.has(key)) continue; seen.add(key); finalUrls.push(item); } The crawler also normalizes host names, removes URL fragments, deduplicates discovered pages, and validates URLs before each fetch. A user submits a URL, so the engine treats every redirect and discovered link as untrusted input. That matters for SSRF protection. It also keeps the page count predictable. The crawl order stays stable even when requests finish in a different order. The report shows the homepage first, then core pages, then sitemap URLs and internal links. Stable output makes a report easier to compare with the previous run. The runner uses a small worker pool The engine has 41 registered checks. Running all of them at once created too much outbound pressure for a Cloudflare-hosted worker. Running them one by one made audits slow. The runner uses six workers. Each worker takes the next check from the registry and runs it with a 30-second timeout. The fetch cache sits on the audit context, so two checks asking for the same URL can share the request. const CHECK_CONCURRENCY = 6; const PER_CHECK_TIMEOUT_MS = 30_000; const ctx = { origin, pages, crawlerConfig: opts.config, fetchCache, }; async function runCheckWorker() { while (true) { const checkIndex = nextCheckIndex++; const check = checkEntries[checkIndex]; if (!check) return; const results = await Promise.race([ check.run(ctx), timeoutAfter(PER_CHECK_TIMEOUT_MS), ]); rawResults[checkIndex] = { checkId: check.id, results, }; } } The real code also converts thrown errors into skip results. A crashed check should not take down the whole report. A slow third-party API should not turn every site into a failure. This design gives the audit a bounded shape: six checks can work at the same time one check gets 30 seconds the worker has its own execution limit a repeated fetch can use the per-audit cache Those numbers are engineering limits, not claims about Google’s review process. The audit tells you what this run verified within its budget. Runtime failures must not become site failures Cloudflare Workers impose subrequest limits and execution limits. A check may receive a network error, a timeout, or a “too many subrequests” error even when the target site has a valid page. The runner inspects failed results for runtime signals. It changes those results into environment-limited skips: function normalizeRuntimeFailure(result: CheckResult) { if (result.status !== 'fail') return result; const text = `${result.message} ${JSON.stringify(result.details ?? {})}`.toLowerCase(); const runtimeFailure = text.includes('too many subrequests') || text.includes('fetch failed') || text.includes('network error') || text.includes('timeout') || text.includes('aborted'); if (!runtimeFailure) return result; return { ...result, status: 'skip' as const, verificationState: 'environment-limited' as const, message: `The audit runtime could not complete this fetch: ${result.message}`, }; } This is not a perfect classifier. Error messages vary by provider and by failure point. It is still better than showing a red failure for a problem the site owner cannot fix. The report surfaces these cases in a separate verification section. A skipped check does not disappear. It stays visible, with a reason and a suggestion to run the audit again or confirm the item by hand. The score rewards evidence The scoring model uses three tiers: Required checks: 3.5 points each Recommended checks: 2.5 points each Suggested checks: 1.5 points each The pass status adds points. Warnings, failures, skips, and informational notes do not subtract points. The maximum is about 92.5 by design. That last detail surprised me at first. Many product teams reach for a 100-point score because users understand it at a glance. A perfect score would suggest that the audit knows everything about a site. It does not. The audit sees fetched pages, response headers, structured data, selected performance data, and the checks that completed within the runtime budget. A score of 78 with three unverified checks tells a better story than a score of 78 with every check verified. The dashboard shows both the readiness result and the verification coverage. The recommendation list also has a cap. It shows up to seven actionable items, with a maximum of five hard failures. A report with 41 rows can help an engineer. A report that asks a site owner to fix 23 things before lunch creates paralysis. The registry protects the system from missing checks Each check lives in its own module and enters the runner through a central registry. The catalog and the registry get compared at startup. That check catches three boring but expensive mistakes: A check exists in the catalog but never runs. A registry entry uses an ID that the catalog does not know. A check returns no result, leaving a silent hole in the report. I also test the scoring semantics with fixtures. One fixture verifies that a warn does not deduct points. Another verifies that an environment-limited skip does not lower the score. A third verifies that an info note stays out of the warning count. These tests look less exciting than adding another check. They protect the meaning of every report, so they earn their place. A crawl is evidence, not a verdict The engine fetches HTML. It does not pretend to be a full browser. Client-rendered content can remain invisible to an HTML crawler. PageSpeed data comes from a separate API. WHOIS data can carry an advisory note rather than a pass or fail. Manual review still matters for editorial quality and policy questions. That boundary belongs in the product copy. The tool can point to missing pages, broken links, weak metadata, absent headers, and other observable issues. It cannot promise an AdSense approval. Google does not expose a public endpoint that lets a third-party tool reproduce the final publisher decision. The useful promise is smaller: run a repeatable preflight check, show the evidence, keep runtime limits visible, and give the site owner a short list of things to fix next. What I would build next I want to add a login event table and a better audit replay view. The first would separate account creation from later use. The second would show how a score changed after a site owner fixed a page. I would also add a browser-rendered pass for sites that depend on client-side routing. That feature needs its own budget and its own verification state. The report should place rendered pages in a separate category from server-rendered pages. The main lesson from the first version is plain: an audit engine needs to describe its uncertainty. Users can work with a report that says “I could not verify this.” They have a harder time with a confident score built from guesses. References Google AdSense eligibility requirements Google Search Essentials Google guidance on creating helpful, reliable content