Unicode text and normalization
A search box receives two spellings of café. They look identical on screen but only one retrieves the saved record. Follow the characters from input to lookup. Read bytes and encoding first.
On this page
Inspect what the string contains
A Python string contains Unicode code points. One visible letter can be represented by one code point or by a base letter followed by a combining mark. String equality compares the sequence it receives. A font can render two different sequences alike.
Choose a lookup representation
from unicodedata import normalize
saved = "caf\u00e9"
typed = "cafe\u0301"
print(len(saved), len(typed), saved == typed)
def key(text):
return normalize("NFC", text)
labels = {key(saved): "coffee"}
print(labels[key(typed)])The first line prints 4 5 False. The typed string stores the accent separately. NFC gives canonically equivalent sequences the same normalized form. Both calls to key now produce the same spelling and the second line prints coffee. The dictionary needs the same rule at insertion and at lookup.
Decide what may be combined
Change the input to uppercase. NFC alone preserves case, so the lookup fails. Decide explicitly whether the application needs case-insensitive lookup. A catalogue may allow several labels to share a search key. A person's display name may need its original spelling. Store the original alongside the derived key when both purposes matter.
Test the precomposed spelling, the decomposed spelling, an uppercase spelling, and an unrelated word. State the intended match for each before you write assertions.