HTTP and JSON
Separate network transport from response validation and keep examples reproducible.
On this page
The concept
An HTTP response has transport behavior, a status, headers, and a body. A successful connection does not prove that the body has the structure your application needs. decode_payload is a pure parsing boundary for an expected items list, so it can be tested without network access.
fetch_json demonstrates a real standard-library request function but is not called by the default demo. It sets a timeout and reads at most one byte beyond a fixed size limit to detect oversize responses. Callers of a real integration must handle HTTP, connection, and timeout failures deliberately. Trust and validate the destination rather than accepting arbitrary URLs blindly.
Worked example
Extract the example files. From the python_from_zero directory, run the following command in a terminal. On Windows, use py in place of python3.
python3 08_professional_workflow/62_http_and_json/lesson.pyimport json
from urllib.request import Request, urlopen
def decode_payload(payload):
data = json.loads(payload.decode("utf-8"))
if not isinstance(data, dict) or not isinstance(data.get("items"), list):
raise ValueError("response must contain an items list")
return data["items"]
def fetch_json(url, *, max_bytes=100000):
request = Request(url, headers={"Accept": "application/json"})
with urlopen(request, timeout=5) as response:
payload = response.read(max_bytes + 1)
if len(payload) > max_bytes:
raise ValueError("response exceeds size limit")
return decode_payload(payload)
if __name__ == "__main__":
print(decode_payload(b'{"items": ["book", "pen"]}'))Trace the result
- decode_payload decodes UTF-8, parses JSON, and checks the expected top-level structure.
- fetch_json limits waiting time and response size before delegating to the parser.
- The guarded demonstration uses prepared bytes only, so running the lesson sends no network request.
['book', 'pen']Line-by-line walkthrough
Line 1: import json
Load json (JSON serialization and parsing). Importing makes these names available; it is not a call to every function in the module.
Line 2: from urllib.request import Request, urlopen
Import Request, urlopen from urllib.request. This supplies HTTP request construction and transport. The caller can now use the imported names directly.
Lines 5–9: def decode_payload(payload):
Define the function decode_payload with input names payload. Definition stores the operation rather than executing its body. An ordinary call executes a normal function; a coroutine runs when awaited/scheduled, and a generator advances when consumed. Decorators can provide a context-manager or wrapper interface. Indented statements below belong to this operation.
Line 6: data = json.loads(payload.decode("utf-8"))
Evaluate json.loads(payload.decode('utf-8')), then bind or update data with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Lines 7–8: if not isinstance(data, dict) or not isinstance(data.get("items"), list):
Evaluate not isinstance(data, dict) or not isinstance(data.get('items'), list). Execute the indented body if it is truthy; otherwise follow the alternative branch, if present. Branch order and indentation are part of the behavior.
Line 8: raise ValueError("response must contain an items list")
Raise ValueError('response must contain an items list'). Normal execution stops here unless an enclosing handler catches it. A from cause, when present, preserves the reason for a translated error.
Line 9: return data["items"]
Evaluate data['items'] and return that value to the caller. This ends the current function call; subsequent statements on this execution path do not run.
Lines 12–18: def fetch_json(url, *, max_bytes=100000):
Define the function fetch_json with input names url, max_bytes. Definition stores the operation rather than executing its body. An ordinary call executes a normal function; a coroutine runs when awaited/scheduled, and a generator advances when consumed. Decorators can provide a context-manager or wrapper interface. Indented statements below belong to this operation.
Line 13: request = Request(url, headers={"Accept": "application/json"})
Evaluate Request(url, headers={'Accept': 'application/json'}), then bind or update request with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Lines 14–15: with urlopen(request, timeout=5) as response:
Enter the listed context manager(s), run the indented block, and leave each context afterward. The manager defines the cleanup/commit behavior; a database transaction and a file lifetime are not identical contracts.
Line 15: payload = response.read(max_bytes + 1)
Evaluate response.read(max_bytes + 1), then bind or update payload with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Lines 16–17: if len(payload) > max_bytes:
Evaluate len(payload) > max_bytes. Execute the indented body if it is truthy; otherwise follow the alternative branch, if present. Branch order and indentation are part of the behavior.
Line 17: raise ValueError("response exceeds size limit")
Raise ValueError('response exceeds size limit'). Normal execution stops here unless an enclosing handler catches it. A from cause, when present, preserves the reason for a translated error.
Line 18: return decode_payload(payload)
Evaluate decode_payload(payload) and return that value to the caller. This ends the current function call; subsequent statements on this execution path do not run.
Lines 21–22: if __name__ == "__main__":
Enter the program entry point only during direct execution. Importing this module still creates definitions but does not run this guarded workflow.
Line 22: print(decode_payload(b'{"items": ["book", "pen"]}'))
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Common mistake
JSON syntax validity is not schema validity. Do not hide connection failures by returning an indistinguishable empty result.