The Procedure Dry Run: How to Test AI-Written Instructions Before Anyone Depends on Them

Instructions are not proven because they sound orderly. A procedure dry run follows the draft literally in a safe test environment, records every place reality diverges, and repairs the steps before a real user carries the risk.

Fluent Instructions Are Not Yet Proven Instructions

Generative AI is very good at producing the shape of a procedure: a confident introduction, a numbered list, familiar labels, and a clean final step. That shape can hide basic failures. A prerequisite may be missing. A menu name may belong to an older version. Two actions may be in the wrong order. The procedure may assume an administrator role, a regional setting, or a file format that the reader does not have. It may say “you are done” without identifying any result that proves completion.

The NIST Generative AI Profile describes confabulation as confidently presented false content and notes that generated output can also be internally inconsistent or contradict its input. In procedural writing, one invented button or unsupported transition can invalidate every later step. Polishing the verbs does not repair that dependency.

A dry run changes the question from “Does this read well?” to “Can the intended person complete the intended task from the stated starting point, and can we see that the result is correct?” It evaluates the procedure as an executable path. Prose quality still matters, but observed task performance becomes the evidence.

Choose the Lowest-Risk Test Lane

Do not prove a draft by gambling with the live system. Match the test environment to the consequence. A formatting tutorial may only need a copy of a document. A product setup guide may need a test account or sandbox. A data-import procedure needs synthetic records, not a customer list. A recovery runbook may need a tabletop exercise or an isolated replica rather than an actual outage.

Before the run, classify every action as read-only, reversible, externally visible, destructive, privileged, or safety-relevant. Remove real recipients, payments, production data, personal information, and live integrations unless an authorized test plan specifically requires them. Set an explicit stop point before any action that would send, publish, delete, purchase, disclose, deploy, or change access.

The FEMA exercise and improvement-planning model uses controlled, low-risk exercises to evaluate capabilities, then turns findings into corrective actions. The exact rigor will differ for a help article, but the transferable principle is sound: exercise the process where failure is contained, capture what happened, and convert the gap into owned work.

For medical, legal, financial, security, industrial, emergency, or other high-consequence instructions, a casual editorial dry run is not validation. Use the relevant domain owner, formal requirements, and approved test method. If a safe representative environment is unavailable, stop and label the procedure unverified rather than testing it on someone else's risk.

Define the Test Contract Before Touching the Steps

A useful dry run begins with a one-page contract. It prevents the tester from unconsciously changing the task until the draft appears to work. Record:

• Intended user: role, experience, access level, and any relevant assistive technology or device constraints.
• Target outcome: the observable state that counts as success, not merely the last instruction.
• Starting state: account type, tool and version, location, existing data, and required configuration.
• Test data: approved synthetic or disposable inputs and the rule for cleaning them up.
• Evidence: what the tester will inspect to confirm each important checkpoint and the final result.
• Boundaries: actions that are forbidden, conditions that end the run, and the person who can authorize more.

NIST's AI Risk Management Framework Core calls for representative populations in evaluations involving people and for performance to be demonstrated under conditions similar to deployment settings. A procedure test uses the same idea at a smaller editorial scale: define who will act, where they will act, and what “works” means in that context. A result from an administrator on a desktop does not automatically validate instructions for a first-time mobile user.

Freeze the Draft and Inventory the Prerequisites

Save the exact version being tested. If the tester rewrites while executing, the evidence no longer points to one procedure. Give the draft an identifier or timestamp, preserve the starting state, and open a separate observation log.

Then inventory everything that must already be true: account, role, permission, subscription level, software version, device, network access, source file, backup, destination, approval, and expected time. Mark each prerequisite as confirmed, missing, or unknown. Do not let the model fill an unknown with a plausible guess.

Google's current developer documentation guidance for procedures recommends giving readers the information they need to prepare, setting the tool or environment before the action, separating reader decisions into distinct list items, and describing useful results after actions. Those are strong authoring rules. The dry run checks whether the draft actually follows them with the real interface and task.

Execute Literally Without Quietly Helping the Draft

Start from the recorded state and follow one instruction at a time. The tester may observe and record, but should not silently supply knowledge the reader was never given. If a label is wrong, note it before finding the likely control. If the draft omits a value, record the pause before choosing one. If a result appears in a different place, capture both the expected and observed location.

Useful observations include the first point of hesitation, every backtrack, outside search, undocumented permission request, error, workaround, and moment when the tester believes the task is complete. False completion matters: a user who confidently stops with the wrong result can be more exposed than one who sees an obvious error.

Do not grade the tester. GOV.UK's guidance on moderated usability testing emphasizes clear, believable tasks, actual or likely users, and watching whether they can complete the work. It also recommends trying the task and instructions with a colleague before sessions. The practical stance is simple: test the instructions, not the person's intelligence.

Run the Main Path, the Branches, and the Recovery Path

A single successful run only proves that one path worked once. After the main path, choose additional cases according to risk:

1. Cold path: a tester with the intended experience level begins without author coaching.
2. Permission path: use the ordinary role, not the elevated role that makes missing access checks disappear.
3. Input path: try a valid but different size, locale, format, or optional field when the procedure claims to support it.
4. Error path: introduce one foreseeable invalid input and verify that the reader can identify and correct it.
5. Interruption path: pause at a meaningful checkpoint and determine whether the task can resume safely.
6. Recovery path: confirm how to cancel, undo, restore, or escalate before the procedure becomes irreversible.

Do not manufacture exotic failures just to make the checklist longer. Prioritize common variations and failures whose consequences compound. W3C's Web Content Accessibility Guidelines 2.2 require labels or instructions where input is needed and address error identification, correction suggestions, and review or reversal for consequential submissions. Even when WCAG does not directly govern a document, those concepts are a useful reminder that an instruction must help people recognize and recover from mistakes, not only describe the perfect path.

Keep an Expected-Versus-Observed Log

For each important step, record the instruction, expected state, observed state, evidence, deviation, and consequence. Add the workaround only after the original failure is visible. Otherwise a skilled tester can make a broken procedure look successful by building an invisible second procedure in the margins.

Classify findings by effect rather than annoyance. A wording issue slows the reader but preserves the outcome. A blocking issue prevents completion. A misleading issue produces the wrong result. A hazardous issue could cause loss, disclosure, injury, or an unauthorized action. Give each nontrivial finding an owner and a retest condition.

Completion rate, time, backtracks, help requests, and false-completion rate can support repeated testing. They are not universal quality scores. Keep the task, environment, and participant context beside the numbers so a fast expert run is not mistaken for proof that any reader can succeed.

Repair the Procedure Around What Actually Happened

Fix the earliest cause, not only the latest symptom. If step six failed because step two never confirmed a role, add the role requirement before step one. If the tester chose the wrong file because the required columns were unstated, define the input before the upload control. If an action succeeded but the next page loaded slowly, describe the observable completion state instead of prescribing a fragile wait time.

Use one decision or action per numbered step. Put the location before the action, name required values and formats, and state an expected result when it helps the reader navigate. Place a warning immediately before the risky action. Make optional and conditional steps explicit. If a path cannot be safely undone, require review or confirmation before it begins.

High-consequence fields use much stronger validation than ordinary publishing. FDA's human-factors guidance for medical devices, for example, centers intended users, intended uses, use environments, critical tasks, and realistic test conditions. That document is not a general rule for blog instructions, but it illustrates why consequence changes the burden of proof. A style edit is not an acceptable substitute for the applicable validation process.

Worked Example: Importing a CSV Into a Project Workspace

Suppose an AI drafts this procedure: “Open Settings, choose Import, upload your CSV, map the fields, and select Finish. Your tasks will appear in the project.” It is compact, grammatical, and not yet dependable.

The editor creates a disposable workspace and a synthetic file containing five tasks. The test contract specifies a standard member role, the current web version, a UTF-8 CSV, no real names, and success as exactly five tasks appearing in a new test project with the intended owners left blank. Sending invitations, overwriting existing tasks, or importing into a live project ends the run.

Literal execution exposes six gaps. The standard member cannot see Import; a workspace administrator must enable it. The current menu says “Import data,” not “Import.” The first row must contain headings. Dates are interpreted according to the workspace locale. The mapping screen preselects an “assignee email” column that would invite people if real addresses were present. Finally, Finish starts an asynchronous job; the tasks do not appear until the Imports page reports “Completed.”

The repaired procedure moves role and file requirements above the steps, directs the user to a disposable project first, explains the date format, tells the user to leave the assignee field unmapped, and names the Imports page as the completion checkpoint. It adds a branch for duplicate rows and a cleanup step for deleting the test project. The editor reruns from a clean workspace with a second tester. Only after that tester completes the task without coaching does the procedure move to live-data review by the accountable product owner.

The dry run did not prove that every account, locale, file, or future version will work. It produced a bounded claim: this revision worked for these representative conditions, and its remaining limits are visible.

Rerun With a Fresh User and No Coaching

The author is usually the worst final tester because they know what every sentence intended to say. After repairs, reset the environment and give the frozen procedure to someone who resembles the reader but did not help write it. Ask them to work toward the goal and think aloud. Do not point to the right control or translate a confusing step while they are still attempting it.

Digital.gov's usability-test model recommends participants who actually use the application or perform the supported task, scenarios aligned with their goals, and prompts that do not reveal how to complete the task. Record success, abandonment, false completion, time, and assistance. Then ask what the tester expected at each failure point.

Re-run after a meaningful change to the interface, permissions, dependencies, input format, policy, or procedure. Add a review date and owner where change is likely. A procedure that once passed can quietly expire.

The Procedure Dry-Run Checklist

• The intended user, goal, starting state, version, and success evidence are explicit.
• The test uses the lowest-risk representative environment and approved dummy data.
• Prerequisites, permissions, inputs, approvals, and forbidden actions are listed.
• The frozen draft is followed literally, without silent author corrections.
• Main, permission, likely-input, error, interruption, and recovery paths are tested in proportion to risk.
• Expected and observed states, workarounds, consequences, owners, and retest conditions are logged.
• Warnings, confirmations, completion signals, rollback, and escalation appear where needed.
• A fresh representative user reruns the revised procedure from a clean state without coaching.
• Unverified conditions and the next review trigger remain visible to readers and owners.

Use AI to Prepare the Test, Not to Declare That It Passed

AI can help inventory the draft before execution. Give it only information approved for that tool and use a bounded instruction such as:

“Review this procedure as a test planner, not as an executor or approver. Extract the intended user, goal, starting state, prerequisites, permissions, inputs, decisions, actions, expected results, branches, error paths, recovery steps, and stop conditions that are explicitly stated. Mark missing information as unknown. Do not invent interface labels, commands, values, policies, permissions, results, or safety claims. Propose a test log and low-risk dummy inputs for human approval. Do not perform any action.”

The output is a planning aid, not evidence. A model cannot confirm that a control exists by repeating its own draft. It cannot authorize a sandbox, decide whether synthetic data is adequate, or declare a high-consequence procedure safe. Those decisions require observable results and the right human authority.

Keep the Human Owner Visible

Every published procedure should have someone accountable for its scope, evidence, unresolved risks, and review trigger. That owner decides what environment is safe, which results count, who must review, and whether the procedure is ready. The tester supplies observations. The writer makes the path clear. A domain expert handles judgments that exceed editorial competence.

The standard is not perfection across every imaginable condition. It is honest, proportionate evidence. State what was tested, preserve what was not, and make failure recoverable where possible. When a procedure will carry real consequences, do not let fluent AI prose borrow trust from a test that never happened.