Invisible Unicode characters in email, and why they are removed
Unicode includes characters you cannot see that change how text looks: direction overrides that reverse a run of letters, and zero-width characters that take up no space. In an email they can make a line display something other than what it contains. Email Digit removes them from the paragraphs, quotes and tables of your emails before they are sent, while keeping the invisible characters that real Arabic, Persian and Indic spelling depend on.
Two families of invisible characters
Most text never contains either of these, which is why they surprise people when they turn up.
| Family | Examples | What it does |
|---|---|---|
| Direction controls | Right-to-left override (U+202E), left-to-right embedding, the isolate characters | Tell the display to lay out the following characters in a different direction |
| Zero-width characters | Zero-width space (U+200B), word joiner (U+2060) | Occupy a position in the text but draw nothing |
Both have honest uses in typesetting. The problem is that the text a reader sees and the text a computer stores stop being the same thing.
How they disguise a line
The classic example is a file name. Put a right-to-left override in front of gpj.exe at the end of invoice_, and the name displays as invoice_exe.jpg. It looks like a picture. It is a program. In 2021 security researchers showed the same trick making source code read one way to a reviewer and run another way, which is how the technique got its wider name, “Trojan Source”.
Zero-width characters are quieter but used in similar ways:
- Breaking up words. A zero-width space inside a word means a filter looking for that word no longer finds it, while a reader sees nothing unusual.
- Making two strings look identical.
example.comwith an invisible character inside it looks the same on screen as the real thing, but it is a different string. - Tagging copies. Patterns of zero-width characters can mark a piece of text so a leaked copy can be traced back.
How they get into ordinary email
Most senders who have these characters in their email did not put them there on purpose. They arrive by paste: from a word processor, a shared document, a web page, a chat message, or an old template that was itself pasted from somewhere else. Nothing on screen shows them, so they travel from campaign to campaign.
If you want to check a piece of copy yourself, most code editors can search with a regular expression. This one matches the direction controls and the common zero-width characters:
[\u200B\u2060\u202A-\u202E\u2066-\u2069]A hit in text that is meant to be plain English is almost always something you can delete.
What Email Digit removes, and what it keeps
When an email is built in Email Digit, the text of paragraphs, quotes and table cells is cleaned as it is rendered for sending. The direction controls are removed, the zero-width space and word joiner are removed, and so are a few other invisible formatting codes that have no use in a message, such as the soft hyphen. Your recipients get the text you can see.
What it leaves alone matters as much:
- Real right-to-left text. Arabic and Hebrew display correctly without any override characters. Email Digit sets the direction of a block with the standard HTML direction attribute, and isolates embedded left-to-right runs such as prices and web addresses, so removing the overrides costs that content nothing.
- The zero-width joiner and non-joiner. Persian and Arabic spelling relies on the non-joiner to stop letters connecting where they should not, Indic scripts need the joiner to form conjuncts, and many emoji are built from several characters held together by the joiner. Removing them would break correct words to stop a rare trick, so they are kept on purpose.
- Combining marks. Arabic vowel marks and Thai vowel signs are invisible on their own and essential in words. They are not touched.
The same cleaning runs in the other direction too. When a reply arrives, its subject, body and sender name are cleaned before they are stored, so a reply in your inbox displays what it actually says.
Limits worth knowing
- It covers body text, not every field. The cleaning runs on paragraphs, quotes and table cells. Subject lines, headings, list items, button labels and anything you paste as a raw HTML block are not passed through it today, so check those by hand if the copy came from somewhere you do not trust.
- It removes invisible characters, not lookalike letters. A web address that swaps a Latin “a” for a Cyrillic one is made of visible characters and is not changed. Read link destinations as well as link text.
- It is not a spam filter. Cleaning hidden characters stops your own email carrying them by accident. It does not judge what your copy says.
You will probably never notice any of this happening, which is the point. If a character is invisible and changes what a line means, it does not belong in an email. Read more about how Email Digit handles deliverability.