A Valid robots.txt Can Still Block the Wrong URLs: Build a Release Matrix

Turn an AI-written robots.txt into a tested crawl contract with a URL matrix, parser evidence, transport probes, and a named release gate.

An AI assistant can produce a tidy robots.txt file in seconds. Every field may be spelled correctly, the file may open in a browser, and a parser may accept it. None of those observations proves that the right crawler can fetch the right URL. A missing slash can block a similarly named public route. A crawler-specific group can stop inheriting the wildcard rules a reviewer assumed it would inherit. A successful edit can remain invisible to a crawler that is using a cached copy.

The useful question is not “Is this valid robots.txt?” It is “For this exact protocol, host, port, crawler product token, and URL, what decision does the deployed artifact produce?” That question can be turned into a release fixture.

This guide builds one intentionally wrong candidate, one corrected candidate, a fourteen-row user-agent-by-URL matrix, and a transport probe. The local parser is deliberately described as a convenience, not a universal oracle. A production release still needs validation against the crawler implementations that matter. The fixture ends with a named HOLD or RELEASE_FIXTURE result while keeping indexing, authentication, and business approval outside the parser’s claim.

Keep Crawl Control, Indexing, and Access Control Separate

Robots Exclusion Protocol rules communicate which resources a cooperating crawler may access. They do not create an authorization boundary. The file is public, so listing a path can make that path easier to discover. A server must still authenticate and authorize requests to anything confidential.

Crawl control is not deindexing either. Google documents that a disallowed URL can still appear in search results without a snippet when Google learns the URL elsewhere. If the requirement is “do not show this page in search,” the owner needs an indexing mechanism that the target search engine supports and can actually observe. Blocking crawling can prevent the crawler from seeing a page-level directive. If the requirement is “only signed-in staff may read this,” robots.txt is irrelevant to enforcement; use access control.

Treat the three jobs as separate evidence lanes:

A green row in the first lane says nothing automatic about the other two. The synthetic fixture uses paths named “private” and “release” to make the crawl decisions easy to see. In a real system, those resources need independent protection, and secret path names should not be advertised in a public file.

Freeze the Release Contract Before Editing Rules

Start with a small release card that a reviewer can read without interpreting syntax:

• Exact initial robots URL, including scheme, host, and port.
• Exact artifact hash or source revision.
• Crawler product tokens in scope.
• A bounded inventory of representative URLs.
• Expected ALLOWED or DISALLOWED result for every user-agent and URL pair.
• The reason each result is wanted.
• Target-crawler parser or validation tool and its version.
• Retrieval status, redirect chain, media type, encoding, size, and cache headers.
• Named owner and RELEASE or HOLD decision.

Authority matters. Google states that rules served from HTTPS do not automatically govern HTTP, a subdomain does not govern its parent, and a nonstandard port has its own scope. Testing https://www.example.test/robots.txt cannot establish the result for https://example.test/robots.txt or https://example.test:8443/robots.txt.

The URL inventory should include more than the obvious blocked directory. Add the public root, an allowed exception below a blocked prefix, a route whose name merely starts the same way, a query-bearing URL, a differently cased path, public CSS or JavaScript needed for rendering, and every specific crawler group. The matrix is the contract. The candidate file is only one proposed implementation.

Read the Protocol Rules Before Reading the Candidate

RFC 9309 defines the protocol baseline. A crawler finds a group by case-insensitively matching its product token. If multiple groups match the same token, their rules are combined. The wildcard group is a fallback when no specific group matches; it is not automatically merged into a specific group. That last point causes a common review error: adding one narrow group can silently remove all wildcard restrictions for that crawler.

Within the applicable group, path matching begins at the first octet. The most specific matching rule is the one with the most octets. An equivalent allow and disallow rule should favor allow. Path comparison should be case-sensitive. Reserved and non-ASCII characters have percent-encoding rules that deserve dedicated fixtures when a site uses them. The protocol also defines the wildcard and end-of-line marker, but a release matrix should test only patterns the policy actually needs.

RFC 9309 requires the file at the lowercase root path /robots.txt, encoded as UTF-8 and served as text/plain. It describes successful, unavailable, and unreachable retrieval outcomes; recommends following at least five redirects; permits caching; and says a cached copy generally should not be used beyond twenty-four hours unless the file is unreachable. Its 500 KiB rule is a lower bound on what compliant parsers must be able to process, not a universal maximum file size.

Crawler behavior can be more specific. Google documents a 500 KiB limit, generally caches for up to twenty-four hours, lets Cache-Control influence lifetime, treats ordinary 4xx responses as no crawl restrictions, gives 429 special treatment, and describes a staged response to 5xx failures using retries and the last good copy when available. Those operational details should be labeled Google behavior, not rewritten as universal RFC behavior.

Let the Wrong Candidate Fail Usefully

The first candidate is syntactically plausible:

User-agent: *
Disallow: /preview
Disallow: /private/
Disallow: /assets/
Allow: /preview/public/

User-agent: ReleaseBot
Disallow: /release/

Sitemap: https://fixture.invalid/sitemap.xml

It contains three policy defects. The prefix /preview also catches /preview-notes and /preview?mode=public. Public rendering assets are blocked. ReleaseBot has a specific group, so the wildcard preview and private rules no longer apply to it; meanwhile, its intended public exception below /release/ is absent.

The corrected file narrows the preview directory and repeats the common rules inside the specific group:

User-agent: *
Disallow: /preview/
Allow: /preview/public/
Disallow: /private/

User-agent: ReleaseBot
Disallow: /preview/
Allow: /preview/public/
Disallow: /private/
Disallow: /release/
Allow: /release/public/

Sitemap: https://fixture.invalid/sitemap.xml

The sitemap line is useful crawler metadata, but it is not an allow or disallow rule in the core protocol. Test the sitemap URL independently. Do not let a parser’s decision about path access stand in for fetching, parsing, or reconciling the sitemap.

Put Expected Decisions in Data

The fixture records complete URLs rather than loose path fragments. That keeps authority visible and makes each row reviewable before any parser runs.

[
  {
    "id": "generic-root",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/",
    "expected": true,
    "reason": "No applicable rule blocks the public root."
  },
  {
    "id": "generic-preview-directory",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/preview/draft.html",
    "expected": false,
    "reason": "The preview directory is outside the crawl contract."
  },
  {
    "id": "generic-preview-public-exception",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/preview/public/index.html",
    "expected": true,
    "reason": "The longer public exception overrides the preview-directory rule."
  },
  {
    "id": "generic-similar-prefix",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/preview-notes",
    "expected": true,
    "reason": "A similarly named public route is not inside /preview/."
  },
  {
    "id": "generic-query-on-public-route",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/preview?mode=public",
    "expected": true,
    "reason": "The public /preview route is not the /preview/ directory."
  },
  {
    "id": "generic-private",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/private/account",
    "expected": false,
    "reason": "The private path is outside the crawl contract, independently of authentication."
  },
  {
    "id": "generic-rendering-asset",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/assets/app.css",
    "expected": true,
    "reason": "Public rendering assets remain crawlable."
  },
  {
    "id": "generic-path-case",
    "userAgent": "OtherBot/1.0",
    "url": "https://fixture.invalid/Preview/draft.html",
    "expected": true,
    "reason": "Path matching is case-sensitive for this contract."
  },
  {
    "id": "releasebot-preview-directory",
    "userAgent": "ReleaseBot/1.0",
    "url": "https://fixture.invalid/preview/draft.html",
    "expected": false,
    "reason": "ReleaseBot needs the preview rule repeated in its specific group."
  },
  {
    "id": "releasebot-preview-public-exception",
    "userAgent": "ReleaseBot/1.0",
    "url": "https://fixture.invalid/preview/public/index.html",
    "expected": true,
    "reason": "ReleaseBot retains the reviewed public exception."
  },
  {
    "id": "releasebot-private",
    "userAgent": "ReleaseBot/1.0",
    "url": "https://fixture.invalid/private/account",
    "expected": false,
    "reason": "The specific group must repeat the private-path rule."
  },
  {
    "id": "releasebot-release-directory",
    "userAgent": "ReleaseBot/1.0",
    "url": "https://fixture.invalid/release/build-42",
    "expected": false,
    "reason": "ReleaseBot must not crawl unreleased build material."
  },
  {
    "id": "releasebot-release-public-exception",
    "userAgent": "ReleaseBot/1.0",
    "url": "https://fixture.invalid/release/public/notes.html",
    "expected": true,
    "reason": "The longer public exception is crawlable."
  },
  {
    "id": "releasebot-rendering-asset",
    "userAgent": "ReleaseBot/1.0",
    "url": "https://fixture.invalid/assets/app.css",
    "expected": true,
    "reason": "ReleaseBot can fetch public rendering assets."
  }
]

Expected values belong to the site owner, not the AI assistant and not the parser. Review those values first. If the intended policy changes, change the release card and matrix through review; do not quietly overwrite an expectation with whatever a new candidate happens to return.

Design Rows That Can Disprove the Policy

A matrix made only of blocked paths is weak evidence. Pair every restriction with a nearby URL that must remain available. The preview rule needs a draft below /preview/, an allowed exception below /preview/public/, a similarly named /preview-notes route, and the bare /preview URL with a query. Those neighbors show whether the rule is a directory boundary or an accidental string prefix.

Do the same for crawler groups. Give ReleaseBot one row that only its specific group blocks, one row that a shared restriction should block, one longer exception, and one ordinary public asset. Keep a generic crawler as a control. This exposes the mistaken idea that a specific group inherits the wildcard group. If the site uses repeated groups for the same product token, add a row for every merged restriction.

Use stable case IDs and plain-language reasons. The ID makes a regression easy to retain; the reason lets a non-parser reviewer challenge the expectation. Store full URLs so a scheme, hostname, or port change cannot disappear inside a path-only test. Avoid real customer identifiers, unpublished tokens, or confidential paths. Synthetic examples can preserve the matching shape without copying sensitive values into logs.

The fourteen rows are not a conformance suite. They do not exercise every percent-encoding, wildcard, Unicode, malformed-line, file-size, or user-agent possibility in RFC 9309. Add those families when the production policy relies on them, and derive expected results from the target specification and crawler documentation. A small matrix tied to a real policy is stronger release evidence than a large collection of unrelated parser trivia.

Use a Local Harness Without Calling It the Crawler

Google publishes the C++ parser used by Googlebot and includes a command-line matcher. That is the appropriate crawler-specific local oracle for Google when the build can be pinned and reproduced. The environment used for this article had no Bazel, CMake, or C/C++ compiler, so it could not build that official implementation.

The runnable example therefore pins robots-parser 3.0.1. It is a third-party teaching and regression dependency. Its result can catch changes between the two candidate files and keep the matrix executable, but it cannot certify Googlebot, Bingbot, another crawler, or future parser behavior. Before production, run the same reviewed matrix through the target crawler’s official parser, report, or validation surface. Record disagreements rather than choosing the convenient green result.

The package also refuses to test under a different Node point release:

{
  "name": "robots-release-fixture",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "engines": {
    "node": "24.19.0"
  },
  "scripts": {
    "test": "node --test matrix.test.mjs transport-probe.test.mjs"
  },
  "dependencies": {
    "robots-parser": "3.0.1"
  }
}

Save the generated lockfile, inspect it, and use a locked install in repeat runs. The test keeps the wrong candidate on HOLD by asserting the exact six failing row IDs. It then requires the corrected candidate to agree with all fourteen expected decisions.

import assert from 'node:assert/strict';
import { readFile } from 'node:fs/promises';
import test from 'node:test';
import robotsParser from 'robots-parser';

const requiredRuntime = 'v24.19.0';
assert.equal(
  process.version,
  requiredRuntime,
  'fixture requires ' + requiredRuntime + '; received ' + process.version,
);

const baseUrl = 'https://fixture.invalid/robots.txt';
const cases = JSON.parse(await readFile(new URL('./cases.json', import.meta.url), 'utf8'));

async function evaluate(candidateFile) {
  const text = await readFile(new URL(candidateFile, import.meta.url), 'utf8');
  const policy = robotsParser(baseUrl, text);
  const rows = cases.map((fixture) => {
    const observed = policy.isAllowed(fixture.url, fixture.userAgent);
    return {
      ...fixture,
      observed,
      matches: observed === fixture.expected,
    };
  });

  return {
    candidateFile,
    total: rows.length,
    failures: rows.filter((row) => !row.matches),
    rows,
  };
}

test('the intentionally wrong candidate remains on HOLD', async () => {
  const result = await evaluate('./candidate-wrong.txt');
  const failureIds = result.failures.map((row) => row.id);

  assert.deepEqual(failureIds, [
    'generic-similar-prefix',
    'generic-query-on-public-route',
    'generic-rendering-asset',
    'releasebot-preview-directory',
    'releasebot-private',
    'releasebot-release-public-exception',
  ]);

  console.log(JSON.stringify({
    candidate: result.candidateFile,
    decision: 'HOLD',
    passed: result.total - result.failures.length,
    failed: result.failures.length,
    failureIds,
  }));
});

test('the corrected candidate matches the complete release matrix', async () => {
  const result = await evaluate('./candidate-correct.txt');
  assert.equal(result.failures.length, 0, JSON.stringify(result.failures, null, 2));

  console.log(JSON.stringify({
    candidate: result.candidateFile,
    decision: 'RELEASE_FIXTURE',
    passed: result.total,
    failed: 0,
  }));
});

The intentional HOLD matters. A sample in which every candidate passes teaches a reviewer to equate test completion with approval. Here, Node’s test process exits successfully because the assertions accurately detect a bad policy and a corrected policy. The candidate decision remains visible inside the evidence.

The six failures also diagnose different classes of mistake. Three come from overly broad wildcard rules: the similar prefix, the query-bearing public route, and the asset directory. Three come from the specific ReleaseBot group: two wildcard restrictions disappeared, and the longer public release exception was never added. The fixture does not merely say “wrong”; it identifies which contract rows need attention.

Probe Retrieval Separately From Rule Matching

A parser receives text. A crawler first has to retrieve that text, possibly through redirects and caches. Testing only a working-tree file misses a wrong-case deployment path, an HTML error page returned with status 200, a stale redirect, an oversized file, invalid UTF-8, or a 503 that changes crawler behavior.

The transport probe begins only at an exact lowercase root /robots.txt URL. It follows and records up to five redirects, retains the initial authority, classifies the final status using the RFC 9309 vocabulary, validates successful bytes as UTF-8 text/plain, applies a conservative 500 KiB release limit for this Google-oriented contract, and reports cache headers. It does not parse rules, declare a URL indexed, or emulate a crawler’s retry schedule.

import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';

const MAX_ROBOTS_BYTES = 500 * 1024;

function classifyStatus(status) {
  if (status >= 200 && status <= 299) return 'successful';
  if (status >= 400 && status <= 499) return 'unavailable';
  if (status >= 500 && status <= 599) return 'unreachable';
  return 'other';
}

function validateInitialUrl(value) {
  const url = new URL(value);
  if (url.pathname !== '/robots.txt' || url.search || url.hash) {
    throw new Error('initial URL must be the exact lowercase root path /robots.txt');
  }
  if (url.protocol !== 'http:' && url.protocol !== 'https:') {
    throw new Error('transport probe supports only HTTP and HTTPS');
  }
  return url;
}

export async function inspectRobotsTransport(value, options = {}) {
  const initialUrl = validateInitialUrl(value);
  const fetchImpl = options.fetchImpl ?? globalThis.fetch;
  const maxRedirects = options.maxRedirects ?? 5;
  const hops = [];
  let currentUrl = initialUrl;
  let redirects = 0;

  for (;;) {
    let response;
    try {
      response = await fetchImpl(currentUrl, {
        redirect: 'manual',
        headers: { 'user-agent': 'RobotsReleaseFixture/1.0' },
      });
    } catch (error) {
      return {
        initialUrl: initialUrl.href,
        initialAuthority: initialUrl.origin,
        rfcAccessClass: 'unreachable',
        releaseGate: 'HOLD',
        redirects,
        hops,
        networkError: error instanceof Error ? error.message : String(error),
        note: 'This probe records transport evidence; it does not predict indexing.',
      };
    }

    const location = response.headers.get('location');
    hops.push({
      url: currentUrl.href,
      status: response.status,
      location,
      contentType: response.headers.get('content-type'),
      cacheControl: response.headers.get('cache-control'),
    });

    if (response.status >= 300 && response.status <= 399 && location) {
      redirects += 1;
      if (redirects > maxRedirects) {
        return {
          initialUrl: initialUrl.href,
          initialAuthority: initialUrl.origin,
          rfcAccessClass: 'other',
          releaseGate: 'HOLD',
          redirects,
          hops,
          error: 'redirect limit exceeded (' + maxRedirects + ')',
          note: 'This probe records transport evidence; it does not predict indexing.',
        };
      }
      currentUrl = new URL(location, currentUrl);
      continue;
    }

    const bytes = new Uint8Array(await response.arrayBuffer());
    const rfcAccessClass = classifyStatus(response.status);
    const mediaType = response.headers
      .get('content-type')
      ?.split(';', 1)[0]
      .trim()
      .toLowerCase();
    let utf8 = false;
    try {
      new TextDecoder('utf-8', { fatal: true }).decode(bytes);
      utf8 = true;
    } catch {
      utf8 = false;
    }

    const successfulFileChecks = {
      exactMediaType: mediaType === 'text/plain',
      utf8,
      within500KiB: bytes.byteLength <= MAX_ROBOTS_BYTES,
    };
    const releaseGate = rfcAccessClass === 'successful'
      && Object.values(successfulFileChecks).every(Boolean)
      ? 'ELIGIBLE_FOR_MATRIX'
      : 'HOLD';

    return {
      initialUrl: initialUrl.href,
      initialAuthority: initialUrl.origin,
      finalUrl: currentUrl.href,
      finalStatus: response.status,
      rfcAccessClass,
      releaseGate,
      redirects,
      byteLength: bytes.byteLength,
      successfulFileChecks,
      hops,
      note: 'This probe records transport evidence; it does not predict indexing.',
    };
  }
}

const invokedPath = process.argv[1]
  ? pathToFileURL(resolve(process.argv[1])).href
  : '';

if (invokedPath === import.meta.url) {
  if (process.version !== 'v24.19.0') {
    throw new Error('fixture requires v24.19.0; received ' + process.version);
  }
  const target = process.argv[2];
  if (!target) {
    throw new Error('usage: node transport-probe.mjs https://example.com/robots.txt');
  }
  const report = await inspectRobotsTransport(target);
  console.log(JSON.stringify(report, null, 2));
  if (report.releaseGate === 'HOLD') process.exitCode = 1;
}

The companion test uses loopback servers, not a public host. It verifies a clean 200 response, one redirect, 404, 503, an HTML body mislabeled by its route as robots.txt, and wrong-case or nested initial paths.

import assert from 'node:assert/strict';
import { createServer } from 'node:http';
import { readFile } from 'node:fs/promises';
import test from 'node:test';
import { inspectRobotsTransport } from './transport-probe.mjs';

const requiredRuntime = 'v24.19.0';
assert.equal(
  process.version,
  requiredRuntime,
  'fixture requires ' + requiredRuntime + '; received ' + process.version,
);

const correctedCandidate = await readFile(
  new URL('./candidate-correct.txt', import.meta.url),
  'utf8',
);

async function withServer(handler, callback) {
  const server = createServer(handler);
  await new Promise((resolve, reject) => {
    server.once('error', reject);
    server.listen(0, '127.0.0.1', resolve);
  });
  const address = server.address();
  assert.ok(address && typeof address === 'object');

  try {
    await callback('http://127.0.0.1:' + address.port);
  } finally {
    await new Promise((resolve, reject) => {
      server.close((error) => error ? reject(error) : resolve());
    });
  }
}

test('transport probe records a successful UTF-8 text file', async () => {
  await withServer((request, response) => {
    if (request.url !== '/robots.txt') {
      response.writeHead(404).end('missing');
      return;
    }
    response.writeHead(200, {
      'content-type': 'text/plain; charset=utf-8',
      'cache-control': 'max-age=300',
    }).end(correctedCandidate);
  }, async (origin) => {
    const report = await inspectRobotsTransport(origin + '/robots.txt');
    assert.equal(report.rfcAccessClass, 'successful');
    assert.equal(report.releaseGate, 'ELIGIBLE_FOR_MATRIX');
    assert.deepEqual(report.successfulFileChecks, {
      exactMediaType: true,
      utf8: true,
      within500KiB: true,
    });
  });
});

test('transport probe follows and records one redirect', async () => {
  await withServer((request, response) => {
    if (request.url === '/robots.txt') {
      response.writeHead(302, { location: '/current-robots.txt' }).end();
      return;
    }
    if (request.url === '/current-robots.txt') {
      response.writeHead(200, { 'content-type': 'text/plain' }).end(correctedCandidate);
      return;
    }
    response.writeHead(404).end('missing');
  }, async (origin) => {
    const report = await inspectRobotsTransport(origin + '/robots.txt');
    assert.equal(report.releaseGate, 'ELIGIBLE_FOR_MATRIX');
    assert.equal(report.redirects, 1);
    assert.deepEqual(report.hops.map((hop) => hop.status), [302, 200]);
  });
});

test('transport probe holds 404, 503, and wrong-media-type observations', async (t) => {
  for (const fixture of [
    { name: '404', status: 404, type: 'text/plain', rfcClass: 'unavailable' },
    { name: '503', status: 503, type: 'text/plain', rfcClass: 'unreachable' },
    { name: 'HTML 200', status: 200, type: 'text/html', rfcClass: 'successful' },
  ]) {
    await t.test(fixture.name, async () => {
      await withServer((request, response) => {
        response.writeHead(fixture.status, { 'content-type': fixture.type }).end('fixture');
      }, async (origin) => {
        const report = await inspectRobotsTransport(origin + '/robots.txt');
        assert.equal(report.rfcAccessClass, fixture.rfcClass);
        assert.equal(report.releaseGate, 'HOLD');
      });
    });
  }
});

test('transport probe rejects a non-root or wrong-case initial path', async () => {
  await assert.rejects(
    inspectRobotsTransport('https://fixture.invalid/Robots.txt'),
    /exact lowercase root path/,
  );
  await assert.rejects(
    inspectRobotsTransport('https://fixture.invalid/nested/robots.txt'),
    /exact lowercase root path/,
  );
});

Install and run the fixture in its own directory:

npm install --ignore-scripts
node --test matrix.test.mjs transport-probe.test.mjs
node transport-probe.mjs https://staging.example.test/robots.txt

The first install creates a lockfile. Subsequent evidence runs should use npm ci with that reviewed lock. Do not point the live transport command at a host you do not own or administer. A transport report marked ELIGIBLE_FOR_MATRIX only means that the successful file passed the byte-level checks; feed the exact fetched artifact into the reviewed parser matrix next.

Interpret Retrieval Outcomes Without Flattening Them

A 404 and a 503 are both HOLD conditions for this deployment workflow, but they are not equivalent protocol observations. Under RFC 9309, a 4xx-style unavailable file can mean the crawler may access resources. An unreachable file, represented by server or network failure, has a complete-disallow baseline. Google’s documented handling adds its own timing, cached-copy, 429, and long-failure behavior. Preserve the raw status and target-crawler interpretation instead of summarizing both as “robots failed.”

Redirects need the same discipline. Record every hop, the initial authority, and the final bytes. RFC 9309 says rules reached through an accepted redirect are followed in the context of the initial authority. The local probe records that initial authority but does not prove how a target crawler will handle a cross-host chain, a missing Location field, six redirects, or a cached earlier response.

After deployment, retrieve every authority in scope independently. Compare the live bytes with the approved artifact, not merely with a rendered browser view. Re-run the user-agent matrix against those bytes. Then perform the crawler-specific validation required for production. If the crawler caches robots.txt, record the expected observation window and avoid claiming immediate effect.

Turn the Evidence Into a Release Decision

Use narrow language:

RELEASE is scoped. It means the named artifact produced the expected crawl decisions for the named crawlers and authorities under the recorded tools and observations. It does not mean every crawler will cooperate, a page will disappear from search, confidential data is protected, or future routes automatically fit the policy.

Reopen review when routes, hostnames, protocols, ports, crawler tokens, parser behavior, redirect logic, caching headers, authentication boundaries, or sitemap structure change. Add each discovered near-miss to the matrix before repairing the candidate so the failure remains reproducible.

Keep AI Inside the Evidence Boundary

AI can draft a candidate file, propose matrix rows, format parser output, and point out duplicate groups. It should not decide which URLs are confidential, whether a public asset may be blocked, which crawler matters, or whether an indexing requirement has been met. Those are owner decisions backed by system and search evidence.

Do not let an assistant “fix” a failing test by flipping expected booleans, deleting inconvenient rows, substituting a different user agent, or relabeling HOLD. Review the matrix as policy, the candidate as configuration, and the output as evidence. If the assistant changes any of them, inspect the diff and rerun from the locked fixture.

The durable habit is simple: write the desired crawl decisions before the rules, make the bad candidate fail in named ways, test the corrected candidate locally, verify the deployed transport, and ask each important crawler its own question. Valid syntax is the start of the review, not the release decision.

Primary Sources and Scope

• RFC 9309, Robots Exclusion Protocol defines groups, matching, encoding, retrieval outcomes, redirects, caching, parser capacity, and the warning that robots.txt is not access control.
• Google’s robots.txt interpretation documents Google-specific authority scope, user-agent selection, rule matching, status handling, caching, format, and size behavior.
• Google’s create and test guidance describes root placement and points developers to Google’s open-source parser.
• Google’s robots.txt introduction distinguishes crawl management from keeping a page out of Google.
• Google’s open-source robots.txt parser documents its production relationship, local command-line matcher, build routes, and the limits of what the parser covers.
• RFC 9110, HTTP Semantics and RFC 9111, HTTP Caching supply the HTTP and caching concepts referenced by RFC 9309.
• Node.js 24.19.0 test runner, HTTP, and global APIs define the executable fixture machinery.

Sources and the local fixture were checked on September 1, 2026. The recorded environment is Node.js v24.19.0 with robots-parser 3.0.1 and synthetic fixture.invalid URLs. The third-party parser is not presented as a standards authority or crawler emulator. The bench does not certify Googlebot, another crawler, indexing, removal, authentication, authorization, confidentiality, production routing, CDN cache state, or future parser behavior. Production RELEASE requires the named crawler-specific validation and live-authority evidence described above.