Bytes and encoding
Explain why a string's character count can differ from its encoded byte count.
On this page
The concept
str represents Unicode text; bytes represents byte values. Encoding turns text into bytes according to a named scheme. Decoding interprets bytes as text. UTF-8 uses a variable number of bytes per code point, so an accented word makes the distinction visible.
The example uses café as data while all instructions remain in English. Its visible accented letter is stored as one code point here and two UTF-8 bytes. A different Unicode representation could use a combining sequence, so code points and visible symbols are not universally identical.
Worked example
text = "café"
payload = text.encode("utf-8")
print(type(text).__name__, type(payload).__name__)
print("characters:", len(text), "bytes:", len(payload))
restored = payload.decode("utf-8")
print(restored)
print(restored == text)Trace the result
- Compare the names of the two types before examining lengths.
- The source string has four code points while its UTF-8 payload has five bytes.
- Decoding with the same encoding restores an equal string.
str bytes
characters: 4 bytes: 5
café
TrueLine-by-line walkthrough
Line 1: text = "café"
Evaluate 'café', then bind or update text with the result. This literal is prepared example data. Assignment itself does not imply a deep copy.
Line 2: payload = text.encode("utf-8")
Evaluate text.encode('utf-8'), then bind or update payload with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Line 3: print(type(text).__name__, type(payload).__name__)
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Line 4: print("characters:", len(text), "bytes:", len(payload))
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Line 5: restored = payload.decode("utf-8")
Evaluate payload.decode('utf-8'), then bind or update restored with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Line 6: print(restored)
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Line 7: print(restored == text)
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Common mistake
Guessing an encoding or ignoring decoding errors can silently lose data. Know the source encoding and define a policy for malformed bytes.