Unicode Inspector

Unicode: What's Actually In That String

Strings look simple. They're not. A character on screen might be:

• A single code point.
• Multiple code points combined (base + combining marks).
• Multiple grapheme clusters (emoji with skin tone modifiers, family emoji built from ZWJ sequences).
• Encoded differently in different normalization forms.
• Visually identical to a different character (homoglyph).
• Contain invisible characters that affect comparison but not display.

The Unicode Inspector shows everything: code points, bytes, categories, names, and flags the invisible stuff that quietly breaks software.

Unicode 101

Code points

Unicode assigns numbers (code points) to characters. The space is 21 bits (max U+10FFFF). Code points written as U+XXXX (4-6 hex digits).

• U+0041 = 'A'
• U+00E9 = 'é'
• U+4E2D = '中'
• U+1F600 = '😀'

Some code points are unassigned, some are non-characters, some are private use (U+E000–U+F8FF, U+F0000–U+10FFFD).

Planes

The 21-bit space is divided into 17 planes of 65,536 code points each:

99% of text lives in the BMP. Emoji are in plane 1.

Encodings

• UTF-8: 1-4 bytes per code point. ASCII-compatible. Web standard.
• UTF-16: 2 or 4 bytes per code point. JavaScript/Java/Windows internal. Surrogate pairs for non-BMP.
• UTF-32: 4 bytes always. Fixed-width. Rarely used.

Categories

Every code point has a category:

Categories matter for: identifier validation, word-break logic, regex \p{L} etc.

Invisible Characters Cheat Sheet

If you paste suspicious text and any of these show up, you've found your bug.

Common Real-World Encoding Bugs

"Café" mojibake

You see café where café should be. Cause: UTF-8 bytes interpreted as Latin-1. The byte 0xC3 0xA9 (é in UTF-8) renders as é in Latin-1.

Fix: declare charset=utf-8 consistently throughout pipeline (HTTP headers, HTML meta, DB column types, file encoding).

Username dedup fails

Database has both alice and alice, visually identical. Inspector reveals one has trailing U+200B (ZWSP).

Fix: normalize on input (NFC), strip control characters, trim whitespace.

Emoji counts wrong

'family: 👨‍👩‍👧'.length returns 14 in JS (UTF-16 code units), but you wanted "11 chars."

Fix: [...new Intl.Segmenter().segment(s)].length returns 11 (grapheme clusters).

Search doesn't find content

User searches for "café" (NFC); content stored as "café" (NFD). String#includes is byte-based; finds nothing.

Fix: normalize both before comparing, a.normalize('NFC').includes(b.normalize('NFC')).

File won't open

File "report.pdf" copied from email. Inspector reveals trailing NBSP, actual filename is "report.pdf" + U+00A0.

Fix: trim non-ASCII whitespace from filenames before saving.

CSV parser breaks

First line starts with "id,name", the BOM (U+FEFF) appears as bytes in some parsers.

Fix: strip leading BOM during CSV ingestion.

Normalization

Unicode has multiple ways to represent the same visual character:

Both render identically; 'é' === 'é' could be false.

Forms

Compatibility differences

NFKC maps compatibility characters to their canonical form:

• ① (U+2460) → 1
• fi (U+FB01) → fi
• ² (U+00B2) → 2
• Ⅻ (U+216B) → XII

Useful for search ("find documents with 2 footnotes" matches both '2' and '²') but lossy.

When to normalize

• User input entering a database (NFC).
• Search indexing (NFKC for fuzzy match).
• File system operations (NFC on most; NFD on macOS).
• Identity comparison (NFC both sides).

JavaScript: s.normalize('NFC').

Counting Characters Correctly

Three different counts

Which to use

• For storage size: byte count (new TextEncoder().encode(s).length).
• For string operations (slicing by character): code points ([...s]).
• For user-visible count (twitter/textarea limits, cursor movement): grapheme clusters (Intl.Segmenter).

Old code uses .length everywhere and gets random emoji bugs.

Homoglyphs

Different code points that render identically:

Threats

• Phishing domains: аpple.com (Cyrillic) vs apple.com (Latin).
• Spoofed usernames: register bаnk_admin to impersonate bank_admin.
• Code injection: function names that look like standard library but aren't.

Defenses

• Domain registries enforce same-script rules.
• IDNA Punycode encoding (xn--80ak6aa92e.com for Cyrillic apple.com) lets users see the difference.
• Username validation can reject mixed-script identifiers.

Working with Code Points in JavaScript

Iterate by code point

for...of iterates code points, not code units. for (let i = 0; i < s.length; i++) iterates code units (would split surrogate pairs).

Get code point

Use codePointAt, not charCodeAt, for emoji.

Build from code point

Regex with Unicode

The u flag enables Unicode features. Without it, . doesn't match astral code points correctly.

Tools and Libraries

Detection / inspection

• This tool, fast inspection.
• unicode-properties (npm), code-level category lookups.
• grapheme-splitter (npm), pre-Intl.Segmenter grapheme handling.

Normalization

• JS built-in: s.normalize(form).
• unorm (npm), for older JS environments.

Bidi / Trojan Source defense

• VS Code: warns since 1.62 (2021).
• ESLint: no-bidi-characters.
• Rust / Go / GCC: warn on bidi in source.

Linters for invisible characters

• hidden-character-detector and similar.
• Pre-commit hook scanning source files for U+2028, U+2029, U+200B, U+FEFF, U+202A–E.

Testing for Unicode Safety

Standard test strings

• NFD vs NFC: 'café' encoded both ways.
• Surrogate pair: '😀'.
• ZWJ sequence: '👨‍👩‍👧'.
• Combining: 'a\u0300' (a + grave).
• RTL: 'مرحبا' (Arabic).
• Mixed direction: 'Hello مرحبا World'.
• Invisible: 'a\u200Bb'.

Run your forms / search / DB through these. If you can't, you have Unicode bugs latent.

Privacy

The inspector uses pure JavaScript: codePointAt, fromCodePoint, TextEncoder (for UTF-8 bytes), Intl.Segmenter (for graphemes), and a built-in Unicode-name table. No network calls. Open DevTools Network during use: zero outbound requests. This is essential for the use case: when you paste a suspicious string into an inspector, you don't want it leaking, the suspicion itself is informative. Strings under inspection often contain partial credentials, internal IDs, suspected attack payloads, log fragments with PII. The work happens in your tab; the text stays there.