Pylerno
English

Bytes and encoding

Explain why a string's character count can differ from its encoded byte count.

On this page

The concept

str represents Unicode text; bytes represents byte values. Encoding turns text into bytes according to a named scheme. Decoding interprets bytes as text. UTF-8 uses a variable number of bytes per code point, so an accented word makes the distinction visible.

The example uses café as data while all instructions remain in English. Its visible accented letter is stored as one code point here and two UTF-8 bytes. A different Unicode representation could use a combining sequence, so code points and visible symbols are not universally identical.

Worked example

python
text = "café"
payload = text.encode("utf-8")
print(type(text).__name__, type(payload).__name__)
print("characters:", len(text), "bytes:", len(payload))
restored = payload.decode("utf-8")
print(restored)
print(restored == text)

Trace the result

  1. Compare the names of the two types before examining lengths.
  2. The source string has four code points while its UTF-8 payload has five bytes.
  3. Decoding with the same encoding restores an equal string.
text
str bytes
characters: 4 bytes: 5
café
True
Line-by-line walkthrough

Line 1: text = "café"

Evaluate 'café', then bind or update text with the result. This literal is prepared example data. Assignment itself does not imply a deep copy.

Line 2: payload = text.encode("utf-8")

Evaluate text.encode('utf-8'), then bind or update payload with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.

Line 3: print(type(text).__name__, type(payload).__name__)

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Line 4: print("characters:", len(text), "bytes:", len(payload))

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Line 5: restored = payload.decode("utf-8")

Evaluate payload.decode('utf-8'), then bind or update restored with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.

Line 6: print(restored)

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Line 7: print(restored == text)

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Common mistake

Guessing an encoding or ignoring decoding errors can silently lose data. Know the source encoding and define a policy for malformed bytes.

Check your understanding

What does this program print?

python
print(len("é".encode("utf-8")))
Choose one answer

Search documentation

Share this page