Pylerno
English

Word analyzer

Define a text-processing contract and produce a stable ranking.

On this page

The concept

The analyzer case-folds text, extracts letter runs, counts occurrences, and ranks the results. Its definition of a word is deliberately limited: the regex excludes digits and underscores but is not a complete linguistic tokenizer. Apostrophes, combining marks, and language-specific boundaries require a more precise policy in a larger application.

Ranking uses descending count and then ascending text for ties. That second key makes repeated runs predictable when frequencies are equal. Empty input returns a valid empty report rather than triggering an unnecessary error.

Worked example

python
import re


def analyze_words(text):
    words = re.findall("[^\\W\\d_]+", text.casefold())
    counts = {}
    for word in words:
        counts[word] = counts.get(word, 0) + 1
    ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))
    return {"total": len(words), "unique": len(counts), "ranking": ranked}


def main():
    text = "Python is useful. I learn Python. I write code, code!"
    report = analyze_words(text)
    print("Total words:", report["total"])
    print("Unique words:", report["unique"])
    for word, count in report["ranking"]:
        print(word, count)
    print("Empty text:", analyze_words(""))


if __name__ == "__main__":
    main()

Trace the result

  1. The regex finds the word units defined by this example.
  2. The dictionary accumulates each word's count.
  3. The tuple sorting key prioritizes larger counts, then alphabetical tie order.
text
Total words: 10
Unique words: 7
code 2
i 2
python 2
is 1
learn 1
useful 1
write 1
Empty text: {'total': 0, 'unique': 0, 'ranking': []}
Line-by-line walkthrough

Line 1: import re

Load re (regular-expression matching). Importing makes these names available; it is not a call to every function in the module.

Lines 4–10: def analyze_words(text):

Define the function analyze_words with input names text. Definition stores the operation rather than executing its body. An ordinary call executes a normal function; a coroutine runs when awaited/scheduled, and a generator advances when consumed. Decorators can provide a context-manager or wrapper interface. Indented statements below belong to this operation.

Line 5: words = re.findall("[^\\W\\d_]+", text.casefold())

Evaluate re.findall('[^\\W\\d_]+', text.casefold()), then bind or update words with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.

Line 6: counts = {}

Evaluate {}, then bind or update counts with the result. The chosen collection determines how later access, membership, and mutation behave. Assignment itself does not imply a deep copy.

Lines 7–8: for word in words:

Obtain each item from words and bind it to word before executing the body. A tuple-shaped target unpacks that item. A loop else, if present, runs only when completion does not use break.

Line 8: counts[word] = counts.get(word, 0) + 1

Evaluate counts.get(word, 0) + 1, then bind or update counts[word] with the result. Assignment itself does not imply a deep copy.

Line 9: ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))

Evaluate sorted(counts.items(), key=lambda item: (-item[1], item[0])), then bind or update ranked with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.

Line 10: return {"total": len(words), "unique": len(counts), "ranking": ranked}

Evaluate {'total': len(words), 'unique': len(counts), 'ranking': ranked} and return that value to the caller. This ends the current function call; subsequent statements on this execution path do not run.

Lines 13–20: def main():

Define the function main with input names no parameters. Definition stores the operation rather than executing its body. An ordinary call executes a normal function; a coroutine runs when awaited/scheduled, and a generator advances when consumed. Decorators can provide a context-manager or wrapper interface. Indented statements below belong to this operation.

Line 14: text = "Python is useful. I learn Python. I write code, code!"

Evaluate 'Python is useful. I learn Python. I write code, code!', then bind or update text with the result. This literal is prepared example data. Assignment itself does not imply a deep copy.

Line 15: report = analyze_words(text)

Evaluate analyze_words(text), then bind or update report with the result. The call finishes before its return value is bound; a failed call does not produce a successful assignment. Assignment itself does not imply a deep copy.

Line 16: print("Total words:", report["total"])

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Line 17: print("Unique words:", report["unique"])

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Lines 18–19: for word, count in report["ranking"]:

Obtain each item from report['ranking'] and bind it to (word, count) before executing the body. A tuple-shaped target unpacks that item. A loop else, if present, runs only when completion does not use break.

Line 19: print(word, count)

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Line 20: print("Empty text:", analyze_words(""))

Evaluate the print arguments from the current state and display them. sep controls separators and end controls the ending when those options are provided. Displaying a value does not return that value from the surrounding function.

Lines 23–24: if __name__ == "__main__":

Enter the program entry point only during direct execution. Importing this module still creates definitions but does not run this guarded workflow.

Line 24: main()

Call main with the displayed arguments. This call may have an effect even when its return value is not assigned.

Common mistake

A program can calculate counts correctly under a poorly specified definition of word. Document tokenization before comparing results with another tool.

Check your understanding

What does this program print?

python
words = "Tea tea".lower().split()
print(len(set(words)))
Choose one answer

Search documentation

Share this page