The Same-Scale Rule: How to Keep AI-Assisted Comparisons Honest

The same-scale rule helps editors catch comparisons that sound decisive while quietly mixing different metrics, units, populations, time windows, methods, denominators, or levels of uncertainty.

AI-assisted drafts are good at producing comparison language.

One option is faster. Another is more popular. Costs fell by 30 percent. A campaign performed better than last year. One model leads the category. A region has twice as many users.

The sentences are fluent, compact, and easy to scan. They can also compare things that were never measured on the same scale.

Perhaps “faster” means average response time for one tool and median completion time for another. Perhaps the 30 percent decline compares one month with an entire quarter. Perhaps “more popular” puts total accounts beside weekly active users. Perhaps two studies use the same label for different outcomes.

The numbers may all be real. The comparison can still be false.

The same-scale rule is simple: before a draft ranks, contrasts, or declares a winner, make sure each side describes the same kind of thing under compatible conditions. If the scales differ, the writer must align them, narrow the claim, or explain why only a limited comparison is possible.

Fluent Prose Can Hide An Uneven Frame

Comparison words create a strong expectation of fairness.

Readers assume that “higher,” “lower,” “faster,” “safer,” and “more effective” refer to a shared measure. A polished paragraph can preserve that expectation even when the source material does not.

AI systems are especially likely to smooth over the mismatch. They can convert inconsistent notes into parallel sentences, place unlike percentages in one list, or invent a broad label that makes different measures appear equivalent. The surface becomes orderly while the evidence remains uneven.

This is not only a math problem. It is an editing problem. A comparison is a claim about the relationship between at least two observations. The relationship needs as much verification as the observations themselves.

Write A Comparison Contract First

Before asking which side is better, write down what a fair comparison would require. Call this the comparison contract.

For each side, record:

• Subject: What exactly is being compared?
• Metric and definition: What outcome was measured, and how was it defined?
• Unit and denominator: Is the result a count, rate, percentage, duration, score, cost per user, or something else?
• Population: Which people, accounts, events, products, or locations were included and excluded?
• Period: When did measurement begin and end?
• Source and method: Who collected the information, and how?
• Uncertainty: Is the result exact, estimated, sampled, rounded, or expressed as a range?

A comparison does not need identical sources in every case. It needs enough compatibility for the specific claim being made. When compatibility is partial, the limits belong beside the conclusion.

Official statistical practice treats this as a core quality issue. The OECD's good statistical practices describe comparable data as relying on common concepts, units, definitions, classifications, and methods, or on clear explanations for differences. That is a useful editorial standard even when the draft is not a formal statistical report.

Start With The Decision, Not The Winner

A weak comparison often begins with “Which one is best?”

That question encourages the draft to find any available difference and turn it into a ranking. A stronger starting point names the decision and the measure that matters.

Instead of:

“Which support channel performed best?”

Try:

“For new customers who contacted support in June, which channel resolved the largest share of cases within 24 hours without a repeat contact in seven days?”

The longer question defines the population, period, outcome, time limit, and quality condition. It may reveal that the available reports cannot answer the question. That is useful information. A responsible draft should not replace a missing comparison with a convenient one.

Check The Denominator Before The Percentage

Percentages look compatible because they use the same symbol. Their denominators may describe entirely different worlds.

Consider:

“Product A converted 18 percent of visitors, while Product B converted 24 percent of trials.”

The sentence appears to show that Product B converts better. But visitors and trials are not the same population. People who begin a trial have already passed several steps that ordinary visitors have not.

For every percentage, ask “percent of what?” Then ask whether the denominator is defined the same way on both sides.

Also watch for:

• percentage change versus percentage-point change;
• events versus unique users;
• eligible customers versus all customers;
• completed responses versus everyone invited;
• per-account cost versus per-active-user cost; and
• market share by revenue versus market share by units.

A denominator is not a footnote. It defines what the rate means.

Align Time Windows And Starting Conditions

A daily average should not quietly compete with a monthly total. A holiday campaign should not be ranked against an ordinary week without seasonal context. A mature product should not be compared with a launch period as if both began from the same baseline.

Record the exact start and end of each observation. Then check:

• Do the windows contain the same number and type of days?
• Did a promotion, outage, policy, release, or unusual event affect one side?
• Are both figures totals, averages, peaks, or end-of-period snapshots?
• Does the comparison use the same baseline?
• Could seasonality explain part of the difference?

If periods cannot be aligned, state the constraint. “June signups were higher than the previous December” is narrower and more honest than “signups are growing” when the draft has only two seasonal snapshots.

Keep Populations And Selection Rules Visible

Two datasets may use the same metric for different groups.

A satisfaction score from paying customers is not directly comparable with a score from every person who visited the site. Completion time for experienced operators does not establish what new users will experience. Results from one region may not travel to another region with different access, language, pricing, or regulation.

Population definitions are substantive. The UK Office for National Statistics notes that time, place, and qualifying conditions shape who belongs inside a population. Changing those conditions changes the estimate, even if the label looks familiar.

Before comparing, write one sentence for each side:

“This result describes ___ who met ___ during ___.”

If the sentences do not describe compatible groups, revise the claim. You may still report both results, but do not use them as clean evidence that one option outperformed the other.

Do Not Treat Shared Labels As Shared Methods

Words such as “engagement,” “accuracy,” “quality,” “retention,” and “success” can hide multiple definitions.

One team may define retention as any return within 30 days. Another may require three active days in the following month. One study may measure accuracy against expert labels. Another may use agreement with a benchmark that contains known errors.

Ask how the value was produced:

• Was it observed, self-reported, modeled, or inferred?
• Was it an average, median, percentile, maximum, or index?
• Were missing cases excluded?
• Did the definition or collection tool change?
• Was the sample large enough for the precision implied by the draft?

A number needs its measure and its uncertainty. NIST guidance emphasizes that a reported measurement result is meaningful together with a clear description of the quantity and the uncertainty associated with it. An editorial comparison should preserve that relationship instead of presenting estimates as perfectly exact facts.

Convert Units Without Converting Meaning

Some mismatches can be fixed mechanically.

Dollars can be converted to euros for a stated date. Minutes can become seconds. Monthly cost can become annual cost when the billing basis is genuinely equivalent.

Other mismatches cannot be repaired through arithmetic. Median time cannot be converted into average time without the underlying distribution. Revenue per paying customer cannot become revenue per registered account without the relevant counts. A five-point satisfaction score cannot be treated as a percentage simply by multiplying it.

AI can perform a unit conversion after the inputs and date are verified. It cannot manufacture the missing relationship that makes two different measures equivalent.

When the conversion changes assumptions, show the assumptions. When the source lacks what the conversion requires, leave the comparison unresolved.

Audit Ranking And Superlative Language

Words such as “best,” “leading,” “fastest,” “most efficient,” and “twice as effective” need a complete frame.

For every ranking, ask:

• Best on which metric?
• Among which alternatives?
• For which population and use case?
• During which period?
• How large is the difference?
• Is the difference larger than the uncertainty or ordinary variation?

If the draft cannot answer those questions, replace the ranking with a descriptive statement.

“Option A had the lowest median response time among the three tools in our June test” is more useful than “Option A is the fastest platform.” The narrower sentence tells readers exactly how far the result can travel.

Ask AI To Find Mismatches, Not Repair Them

AI can be useful as a comparison auditor when the instruction prevents it from filling gaps.

Try:

“List every comparative claim in this draft. For each side, extract the subject, metric and definition, unit and denominator, population, time period, source and method, and stated uncertainty. Mark any field that is missing or incompatible. Do not rank the options, convert values, or infer missing details.”

Then return to the original sources. Confirm every extracted field and resolve the mismatch through evidence, a domain owner, or a narrower claim.

A second prompt can test language:

“Find every word that implies ranking or superiority, including better, worse, faster, safer, leading, more effective, and best. For each word, state the exact comparison frame the draft would need to support it. Do not rewrite the claim.”

This keeps the model in a bounded role: locating pressure points without deciding that unlike evidence is close enough.

Example: A Vendor Comparison

An AI-assisted procurement summary says:

“Vendor A costs 20 percent less and resolves tickets twice as fast as Vendor B.”

The editor checks the contract. Vendor A's cost is listed per active seat before support fees. Vendor B's figure is total invoiced cost per licensed seat. Vendor A reports median time to first response for priority tickets. Vendor B reports average time to final resolution across every ticket.

Nothing in the sentence is comparable yet.

The team requests aligned cost components and exports the same ticket categories for the same quarter. The revised draft separates cost from service speed, names each metric, and leaves one field unresolved because Vendor A does not provide final-resolution time.

The result is less dramatic and more useful for a real purchasing decision.

Example: A Product Experiment

A launch recap says that the new onboarding “improved activation from 12 percent to 19 percent.”

The earlier figure counted every signup and defined activation as completing three tasks within seven days. The later figure excluded unverified accounts and defined activation as completing one task within 14 days.

The two percentages do not show improvement on a shared scale.

The editor reconstructs both cohorts using one definition and one observation window. If that is not possible, the recap reports the two figures as results from different measurement systems and avoids a trend claim.

Refusing the false improvement story does not waste the data. It identifies what the next experiment must measure consistently.

The Final Same-Scale Check

Before publishing a comparison, confirm that:

• the decision or question is defined before a winner is named;
• both sides describe the same subject and outcome;
• units, denominators, and statistical summaries are compatible;
• populations use aligned inclusion and exclusion rules;
• time windows and starting conditions are comparable;
• methods, definitions, and source limitations are visible;
• uncertainty and meaningful size of difference are considered;
• unit conversions do not disguise conceptual mismatches;
• ranking language stays inside the evidence's actual scope; and
• AI has flagged gaps without inventing the facts needed to close them.

An honest comparison does not require perfect symmetry. It requires a clear account of what is shared, what differs, and what conclusion the evidence can actually carry.

Put both sides on the same scale before asking the sentence to decide between them.