Word analyzer
Define a text-processing contract and produce a stable ranking.
On this page
The concept
The analyzer case-folds text, extracts letter runs, counts occurrences, and ranks the results. Its definition of a word is deliberately limited: the regex excludes digits and underscores but is not a complete linguistic tokenizer. Apostrophes, combining marks, and language-specific boundaries require a more precise policy in a larger application.
Ranking uses descending count and then ascending text for ties. That second key makes repeated runs predictable when frequencies are equal. Empty input returns a valid empty report rather than triggering an unnecessary error.
Worked example
import re
def analyze_words(text):
words = re.findall("[^\\W\\d_]+", text.casefold())
counts = {}
for word in words:
counts[word] = counts.get(word, 0) + 1
ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))
return {"total": len(words), "unique": len(counts), "ranking": ranked}
def main():
text = "Python is useful. I learn Python. I write code, code!"
report = analyze_words(text)
print("Total words:", report["total"])
print("Unique words:", report["unique"])
for word, count in report["ranking"]:
print(word, count)
print("Empty text:", analyze_words(""))
if __name__ == "__main__":
main()Trace the result
- The regex finds the word units defined by this example.
- The dictionary accumulates each word's count.
- The tuple sorting key prioritizes larger counts, then alphabetical tie order.
Total words: 10
Unique words: 7
code 2
i 2
python 2
is 1
learn 1
useful 1
write 1
Empty text: {'total': 0, 'unique': 0, 'ranking': []}Line-by-line walkthrough
Line 1: import re
Load re (regular-expression matching). Importing makes these names available; it is not a call to every function in the module.
Lines 4–10: def analyze_words(text):
Define the function analyze_words with input names text. Definition stores the operation rather than executing its body. An ordinary call executes a normal function; a coroutine runs when awaited/scheduled, and a generator advances when consumed. Decorators can provide a context-manager or wrapper interface. Indented statements below belong to this operation.
Line 5: words = re.findall("[^\\W\\d_]+", text.casefold())
Evaluate re.findall('[^\\W\\d_]+', text.casefold()), then bind or update words with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Line 6: counts = {}
Evaluate {}, then bind or update counts with the result. The chosen collection determines how later access, membership, and mutation behave. Assignment itself does not imply a deep copy.
Lines 7–8: for word in words:
Obtain each item from words and bind it to word before executing the body. A tuple-shaped target unpacks that item. A loop else, if present, runs only when completion does not use break.
Line 8: counts[word] = counts.get(word, 0) + 1
Evaluate counts.get(word, 0) + 1, then bind or update counts[word] with the result. Assignment itself does not imply a deep copy.
Line 9: ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))
Evaluate sorted(counts.items(), key=lambda item: (-item[1], item[0])), then bind or update ranked with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Line 10: return {"total": len(words), "unique": len(counts), "ranking": ranked}
Evaluate {'total': len(words), 'unique': len(counts), 'ranking': ranked} and return that value to the caller. This ends the current function call; subsequent statements on this execution path do not run.
Lines 13–20: def main():
Define the function main with input names no parameters. Definition stores the operation rather than executing its body. An ordinary call executes a normal function; a coroutine runs when awaited/scheduled, and a generator advances when consumed. Decorators can provide a context-manager or wrapper interface. Indented statements below belong to this operation.
Line 14: text = "Python is useful. I learn Python. I write code, code!"
Evaluate 'Python is useful. I learn Python. I write code, code!', then bind or update text with the result. This literal is prepared example data. Assignment itself does not imply a deep copy.
Line 15: report = analyze_words(text)
Evaluate analyze_words(text), then bind or update report with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.
Line 16: print("Total words:", report["total"])
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Line 17: print("Unique words:", report["unique"])
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Lines 18–19: for word, count in report["ranking"]:
Obtain each item from report['ranking'] and bind it to (word, count) before executing the body. A tuple-shaped target unpacks that item. A loop else, if present, runs only when completion does not use break.
Line 19: print(word, count)
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Line 20: print("Empty text:", analyze_words(""))
Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.
Lines 23–24: if __name__ == "__main__":
Enter the program entry point only during direct execution. Importing this module still creates definitions but does not run this guarded workflow.
Line 24: main()
Call main with the displayed arguments. This call may have an effect even when its return value is not assigned.
Common mistake
A program can calculate counts correctly under a poorly specified definition of word. Document tokenization before comparing results with another tool.