One Word, Several Byte Sequences: Test AI-Written Unicode Matching Rules

Build a Unicode fixture bench before shipping AI-written matching rules: normalization, case, grapheme clusters, identifiers, confusables, and domain labels.

Ask an AI coding assistant to make usernames “case-insensitive,” deduplicate tags, or compare two customer names, and it may offer a compact recipe: trim the input, convert it to lowercase, and compare the results. The code looks ordinary. It passes tests built from plain ASCII. It may even solve the one example in the prompt.

But trim().toLowerCase() is not a universal Unicode policy. It quietly combines several different questions: which whitespace matters, whether case carries meaning, which locale or protocol supplies the rule, whether canonically equivalent sequences should compare together, whether compatibility characters should collapse, and whether the unit a user sees is one code point or several. A rule suitable for a search box can be destructive in an account identifier. A rule for an internationalized domain label is not a rule for a display name.

The useful review artifact is therefore not a clever normalizing function. It is a fixture bench that makes the intended relationship between strings observable. The bench below starts with product semantics, records exact representations, exercises standards-based transforms, and exposes collisions before a generated rule becomes durable data.

Bench Setup: Name the String Slot

Before choosing an algorithm, give the field a semantic name. “Text” is too broad. The same visible sequence can move through several slots in one product, and each slot can have a different contract.

• Display name: user-facing content whose original spelling, casing, script, marks, and emoji normally need to be preserved. It may be non-unique.
• Username or identifier: a value used to select, address, authorize, or link an account. It needs a documented repertoire, comparison rule, uniqueness boundary, and migration policy.
• Search text: natural-language content where broader matching may be useful, but language, ranking, and false-positive tradeoffs matter.
• Hashtag or label: a user-facing token whose allowed characters and matching behavior may be broader than a programming-language identifier.
• Domain label: an input governed by IDNA-specific mapping and validation, not by a general username or search transform.

Write a one-sentence contract for the chosen slot. For example: “The display name preserves the submitted Unicode string; the account key uses the product’s versioned identifier profile; two accounts may never be merged solely because their display names or confusable skeletons match.” That contract is testable. “Handle Unicode” is not.

Also record where the decision applies. A browser, API, database uniqueness constraint, background importer, and mobile client must not each improvise a different comparison key. If an older client can submit a value the server now rejects, that mismatch belongs on the bench.

Keep Original Input Separate From Comparison Keys

A derived key is not a corrected version of a person’s text. Store or otherwise preserve the original value needed for display and audit, subject to the product’s privacy and retention rules. Put a derived comparison value in a separate field with the profile name and version that produced it.

This separation prevents three common failures. First, a compatibility transform cannot silently rewrite what the user sees. Second, a future library or Unicode-data upgrade can be evaluated without guessing what the original input was. Third, a collision between two derived keys does not erase the distinction between their source strings.

Do not make a comparison key the only recovery record. If a transform is many-to-one, the original cannot be reconstructed from the key. Conversely, do not assume preserving the original makes a comparison rule safe. Uniqueness and authorization still depend on the exact server-side profile, not on whichever representation the interface happens to show.

Build a Fixture Ledger Before Writing the Function

Each fixture should show more than two attractive strings. Record the displayed form, its code-point sequence, its UTF-8 bytes, the expected relation for the named slot, and the reason for that expectation. Keep the expected result independent from the implementation under test; otherwise the test only repeats the code’s assumption.

Displayed fixture	Representation	Expected relation	Rationale to record
é and é	U+00E9 / UTF-8 C3 A9; versus U+0065 U+0301 / 65 CC 81	Canonically equivalent after the chosen canonical-normalization step; not byte-identical before it.	The slot contract explicitly accepts canonical equivalence.
A and Ａ	U+0041 / 41; versus U+FF21 / EF BC A1	Distinct under NFC; compatibility-related under NFKC.	The product must decide whether erasing width compatibility is appropriate for this slot.
👩🏽‍💻	U+1F469 U+1F3FD U+200D U+1F4BB; UTF-8 F0 9F 91 A9 F0 9F 8F BD E2 80 8D F0 9F 92 BB	Several code points, normally one extended grapheme cluster under the default rules.	Cursor movement or a visible-character limit should not assume one code point equals one user-perceived character.
Latin a and Cyrillic а	U+0061 / UTF-8 61; versus U+0430 / D0 B0	Distinct identifiers even if a confusable check raises a review signal.	Visual similarity is not identity equivalence and does not prove intent.

Render the ledger with a font representative of the product, but never use appearance as the only evidence. Copying a string from a UI screenshot loses the exact code points. A debugging view should escape controls and show scalar values and bytes without exposing sensitive production text in ordinary logs.

Exercise 1: Separate Canonical and Compatibility Normalization

Unicode Normalization Forms define two kinds of equivalence. NFC and NFD preserve canonical equivalence while choosing composed or decomposed representations. That is why precomposed é and e followed by U+0301 COMBINING ACUTE ACCENT can become comparable under a canonical-normalization policy even though their original UTF-8 bytes differ.

NFKC and NFKD also apply compatibility mappings. Those transforms can bring together forms such as ASCII A and fullwidth Ａ, or the fi ligature and the two-letter sequence fi. That broader collapse can be appropriate inside a deliberately defined identifier or search profile. It can also erase distinctions someone intended to retain. Do not upgrade from NFC to NFKC because it catches more fixtures; decide what the slot means first.

For every form the implementation claims to use, add positive and negative fixtures. Positive fixtures prove that the intended equivalents meet. Negative fixtures protect distinctions that must remain. Then check idempotence: applying the chosen normalization twice should produce the same output as applying it once.

Exercise 2: Test the Concatenation Boundary

A normalized fragment plus another normalized fragment is not guaranteed to remain normalized after concatenation. Unicode’s normalization annex calls out this boundary explicitly. One simple NFC fixture joins U+0061 LATIN SMALL LETTER A with a separately held U+030A COMBINING RING ABOVE. Each fragment can be NFC on its own, while the combined sequence can compose to U+00E5 LATIN SMALL LETTER A WITH RING ABOVE.

Exercise every place the system assembles text: prefix plus user value, first name plus mark, imported chunks, streaming decoders, autocomplete edits, and database updates that splice substrings. Normalize at the boundary required by the documented algorithm rather than assuming upstream fragments stay safe when joined. Add a test in which a combining mark arrives in a later chunk.

Exercise 3: Make Case Behavior Explicit

Lowercasing for display, full case folding for comparison, ASCII-only case-insensitive matching, and language-tailored search are different operations. A generated helper named normalizeName can conceal which one it chose.

Build a case set that includes plain ASCII plus cases that expose hidden assumptions: German ß, Greek Σ, σ, and final ς, and the dotted and dotless I family I, i, İ, and ı. The expected results must come from the slot’s specified profile and the library mode being invoked. Do not publish a universal Turkish-I rule, and do not infer a user’s language merely from one character.

Keep display casing separate from the comparison key. Test that the original survives a save-and-load cycle. Test the same operation under every runtime locale the application can inherit, then eliminate accidental dependence on process defaults. If the specification calls for case-sensitive matching, a lowercasing step is a bug even if it feels user-friendly.

Exercise 4: Count Grapheme Clusters, Not Just Code Points

A user-perceived character can contain several code points. The default extended-grapheme rules cover combining sequences, emoji modifiers, zero-width-joiner sequences, and regional-indicator pairs, among other cases. They are a practical approximation, not a claim that every language and interface defines “character” identically.

Put é, 👩🏽‍💻, a flag sequence, and adjacent emoji into the bench. Check deletion, cursor movement, truncation, selection, visible-length limits, and preview rendering. A four-code-point emoji sequence should not be cut between its joiner and the following pictograph merely because a generated loop slices by scalar count or UTF-16 code unit.

Word and sentence boundaries need their own tests. Unicode’s default segmentation rules allow documented tailoring, and some languages require dictionary or other locale-aware behavior for useful word boundaries. A search highlighter and an identifier validator therefore should not share one vague “character splitter.” Record which segmentation algorithm and version each feature uses.

Exercise 5: Define an Identifier Profile

For identifier-like slots, write down the allowed repertoire and the transforms in order. Unicode UAX #31 supplies recommended identifier properties such as XID_Start and XID_Continue, together with normalization and stability guidance. Those properties are useful building blocks for identifiers; they are not a universal username product policy.

Application protocols may instead rely on an existing PRECIS profile. RFC 8264 separates preparation, enforcement, and comparison and provides base classes for identifier-like and free-form strings. If a governing protocol already defines a profile, implement that profile with a maintained library. Do not mix pieces of UAX #31, PRECIS, a regular expression, and a database collation into a new undocumented hybrid.

The bench should cover the first character, continuation characters, combining marks, spaces, punctuation, unassigned code points, and code points whose treatment depends on context. Pin the Unicode-data version used by the implementation. Add a fixture for every product-specific addition or exclusion and state why the profile needs it.

Exercise 6: Expose Invisibles and Bidirectional Controls

Some format controls are essential in particular scripts or emoji sequences; others can make two strings look alike while their underlying order or content differs. Removing every invisible character would damage legitimate text. Accepting every control in a security-sensitive identifier without a profile would create a different problem.

Add fixtures containing a zero-width joiner in a valid emoji sequence, an unexpected joiner, directional marks, a bidirectional override, variation selectors, and leading combining marks. In the test report, show escaped code points and both logical and rendered order. Test copy and paste, not only initial keyboard entry.

For display names, the result may be preservation plus safe rendering. For an identifier, the selected profile may reject or contextually restrict some controls. The important point is that a developer can trace the behavior to the slot contract. A blanket “strip nonprinting characters” instruction from an AI assistant is not that contract.

Exercise 7: Treat Confusables as Signals, Not Equality

Unicode UTS #39 defines mechanisms for detecting visually confusable strings, including skeleton-based comparisons and mixed- or whole-script analysis. The standard also explains that the mechanism can be overly inclusive. Typography, language, surrounding text, and the application’s threat model all affect what is suspicious.

Put Latin a and Cyrillic а on the bench, along with legitimate multilingual examples. A matching skeleton can trigger a warning, additional review, or a product-specific naming rule. It must not silently merge accounts, transfer ownership, label a person fraudulent, or turn one string into the other. Confusable detection answers a risk-screening question, not the identity question.

Test the user experience as well as the detector. Can the interface reveal the exact identifier without shaming legitimate script use? Is an appeal or alternate selection path available? Are internal alerts based on the versioned skeleton data, and can the team measure false positives without logging private text unnecessarily?

Exercise 8: Put Domain Labels on Their Own Track

Internationalized domain names have domain-specific rules. Unicode UTS #46 defines compatibility processing used with IDNA and distinguishes Unicode labels from the ASCII-compatible labels used in DNS. For example, within the illustrative domain name bücher.example, a trusted implementation can convert the Unicode label bücher to the A-label xn--bcher-kva.

Do not reproduce that behavior with lowercase, NFKC, and a hand-written Punycode call. Use a maintained IDNA implementation in the mode required by the product and platform, validate every label and the full name, and compare the exact form appropriate to the operation. Current UTS #46 guidance uses nontransitional processing; legacy behavior and platform differences still belong in the fixture matrix.

IDNA processing does not solve visual confusables. A label can be valid under domain-processing rules and still deserve a separate display or spoofing review. Conversely, a string policy for usernames should not inherit domain mappings merely because both values are called “identifiers.”

Turn the Ledger Into Executable Properties

Example fixtures catch known edges. Property and conformance tests catch combinations the author did not imagine. Build the suite in layers:

1. Standards vectors: run the conformance data published for the normalization and segmentation algorithms your library claims to implement.
2. Product fixtures: encode every expected match, expected distinction, allowed input, rejection, and contextual rule from the slot contract.
3. Properties: test idempotence, deterministic output, valid encoding, preservation of the original, and stability across repeated serialization.
4. Generated combinations: vary bases, combining marks, scripts, controls, chunk boundaries, case forms, and grapheme sequences within safe test limits.
5. Cross-layer checks: send the same fixture through browser, API, service, database lookup, importer, and background job.

Add protected-pair tests: values that must never collapse under the chosen policy. For unique identifiers, assert that a collision is handled explicitly rather than allowing the database to choose whichever row happens to win. For search, measure unwanted matches as well as missed ones.

Record the runtime, library, Unicode-data version, profile name, and configuration with each run. “Works on my machine” is particularly weak evidence when two platforms bundle different internationalization libraries or data versions.

Upgrade Without Rewriting Identity

Unicode and internationalization libraries evolve. A data upgrade can admit new code points, alter derived properties, or change segmentation and security data. Treat that as a data migration, not a routine invisible dependency bump.

Before switching, run the old and new profiles over the saved fixture corpus and an authorized, privacy-minimized sample. Count keys that change, previously distinct keys that collide, inputs that become accepted or rejected, and confusable signals that appear or disappear. Never auto-merge accounts because a newer derived key collides.

A reversible deployment can write versioned old and new keys side by side, perform shadow comparisons, and keep reads on the established rule until the differences are understood. Instrument counts by outcome and profile version, not raw private strings. Provide a rollback path that restores the earlier comparison rule without discarding original input or newly created records.

After the change, watch rejection rates, uniqueness conflicts, search-result shifts, support reports, and cross-service disagreement. Observability should tell the team which rule ran and why a decision category changed while avoiding sensitive text in logs.

Give AI a Bounded Job at the Bench

AI can propose fixture families, explain unfamiliar code points, draft property-test scaffolding, compare implementation steps with a cited specification, and turn a confirmed product contract into readable test names. Every suggested example still needs its code points and expected relation checked against the actual standard and library.

AI should not decide that two customer identifiers represent one person, invent a locale from a name, choose a compatibility fold because it produces fewer duplicates, or write a substitute for a maintained IDNA implementation. It also should not call a confusable string malicious. Those are product, protocol, security, and human-context decisions.

The bench passes when the team can explain why each pair matches, differs, warns, or rejects in one named string slot—not when one normalization function makes every fixture equal.

A Small Lab Walkthrough

Suppose an AI assistant generates one helper for display names, login names, and search terms. The helper trims, applies NFKC, lowercases, and stores only the result. ASCII tests pass.

The fixture bench changes the design before production data is committed. Display names retain the submitted value and receive no uniqueness role. Login names use a documented identifier profile with a separate versioned comparison key and protected-pair tests. Search uses its own language-aware behavior and is evaluated for both missed and unwanted results. Domain inputs go through a trusted IDNA path. Confusable skeletons feed a review signal but never account identity.

The é fixtures verify intended canonical equivalence for the login key while preserving original bytes for display. Fullwidth and ligature fixtures reveal whether compatibility folding is actually part of the profile. Turkish-I and sigma fixtures expose default-locale dependencies. The emoji fixture prevents a visible-length limit from cutting a joiner sequence. The concatenation fixture catches a streaming path that normalized chunks but not their combined output.

No single result proves the system universally correct or secure. It does establish something more useful than fluent generated code: a reviewable mapping from product meaning to standards-based behavior, with failures that can be reproduced before users depend on it.

Source and Version Note

Sources were checked on August 24, 2026. The five cited Unicode annexes and technical standards are pinned to their stable Unicode 17.0 versions. The W3C string-matching document was a First Public Working Draft on that date, while RFC 8264 remains an IETF Standards Track RFC. Implementations should record the exact Unicode data and library version they deploy; a current specification does not prove that a bundled runtime implements the same version.

• Unicode Standard Annex #15: Unicode Normalization Forms defines NFC, NFD, NFKC, and NFKD, conformance testing, and normalization boundary behavior.
• Unicode Standard Annex #29: Unicode Text Segmentation specifies default grapheme, word, and sentence boundaries and explains where tailoring may be needed.
• Unicode Standard Annex #31: Unicode Identifiers and Syntax defines recommended identifier properties, profiles, normalization, case, and stability considerations.
• Unicode Technical Standard #39: Unicode Security Mechanisms specifies restriction and confusable-detection mechanisms while documenting their scope and limitations.
• W3C Character Model for the World Wide Web: String Matching is a First Public Working Draft that separates code-point conversion, normalization, case behavior, comparison, and tailoring.
• RFC 8264: PRECIS Framework defines preparation, enforcement, and comparison building blocks for internationalized strings in application protocols.
• Unicode Technical Standard #46: Unicode IDNA Compatibility Processing defines mapping and validation processing used with internationalized domain names.

This fixture lab does not guarantee security, correct internationalization for every language, or compatibility across every platform. It does not replace a governing protocol, qualified security and internationalization review, user research, or the conformance suite for the chosen implementation.