CSV and Text

Count Token Frequencies

Split a sentence into words with .split() and count how many times each word appears. The trace shows counts seeding new words at 1 and incrementing on repeats — the word 'the' appears three times and 'cat' twice.

By hand

Split the text into a word list with .split(), then walk the words with the same seed-or-increment dict pattern from frequency-count (ch05).

naive.py
Replay: real traced execution (multi-file project)
text = 'the cat sat on the mat the cat'
words = text.split()
counts = {}
for w in words:
    if w in counts:
        counts[w] = counts[w] + 1
    else:
        counts[w] = 1
print('RESULT:', {k: counts[k] for k in sorted(counts)})

  1. text ← 'the cat sat on the mat the cat'

    1text = 'the cat sat on the mat the cat'2words = text.split()
    values this step'the cat sat on the mat the cat'text

  2. words ← ['the', 'cat', 'sat', 'on', 'the', 'mat', 'the', 'cat']

    1text = 'the cat sat on the mat the cat'2words = text.split()3counts = {}
    values this step['the', 'cat', 'sat', 'on', 'the', 'mat', 'the', 'cat']words

  3. counts ← {}

    2words = text.split()3counts = {}4for w in words:
    values this step{}counts

  4. w ← 'the'

    3counts = {}4for w in words:5    if w in counts:
    values this step'the'w

  5. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  6. counts ← {'the': 1}

    7    else:8        counts[w] = 19print('RESULT:', {k: counts[k] for k in sorted(counts)})
    values this step{} → {'the': 1}counts

  7. w ← 'cat'

    3counts = {}4for w in words:5    if w in counts:
    values this step'the' → 'cat'w

  8. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  9. counts ← {'the': 1, 'cat': 1}

    7    else:8        counts[w] = 19print('RESULT:', {k: counts[k] for k in sorted(counts)})
    values this step{'the': 1} → {'the': 1, 'cat': 1}counts

  10. w ← 'sat'

    3counts = {}4for w in words:5    if w in counts:
    values this step'cat' → 'sat'w

  11. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  12. counts ← {'the': 1, 'cat': 1, 'sat': 1}

    7    else:8        counts[w] = 19print('RESULT:', {k: counts[k] for k in sorted(counts)})
    values this step{'the': 1, 'cat': 1} → {'the': 1, 'cat': 1, 'sat': 1}counts

  13. w ← 'on'

    3counts = {}4for w in words:5    if w in counts:
    values this step'sat' → 'on'w

  14. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  15. counts ← {'the': 1, 'cat': 1, 'sat': 1, 'on': 1}

    7    else:8        counts[w] = 19print('RESULT:', {k: counts[k] for k in sorted(counts)})
    values this step{'the': 1, 'cat': 1, 'sat': 1} → {'the': 1, 'cat': 1, 'sat': 1, 'on': 1}counts

  16. w ← 'the'

    3counts = {}4for w in words:5    if w in counts:
    values this step'on' → 'the'w

  17. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  18. counts ← {'the': 2, 'cat': 1, 'sat': 1, 'on': 1}

    5if w in counts:6    counts[w] = counts[w] + 17else:
    values this step{'the': 1, 'cat': 1, 'sat': 1, 'on': 1} → {'the': 2, 'cat': 1, 'sat': 1, 'on': 1}counts

  19. w ← 'mat'

    3counts = {}4for w in words:5    if w in counts:
    values this step'the' → 'mat'w

  20. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  21. counts ← {'the': 2, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1}

    7    else:8        counts[w] = 19print('RESULT:', {k: counts[k] for k in sorted(counts)})
    values this step{'the': 2, 'cat': 1, 'sat': 1, 'on': 1} → {'the': 2, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1}counts

  22. w ← 'the'

    3counts = {}4for w in words:5    if w in counts:
    values this step'mat' → 'the'w

  23. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  24. counts ← {'the': 3, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1}

    5if w in counts:6    counts[w] = counts[w] + 17else:
    values this step{'the': 2, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1} → {'the': 3, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1}counts

  25. w ← 'cat'

    3counts = {}4for w in words:5    if w in counts:
    values this step'the' → 'cat'w

  26. if w in counts:

    4for w in words:5    if w in counts:6        counts[w] = counts[w] + 1

  27. counts ← {'the': 3, 'cat': 2, 'sat': 1, 'on': 1, 'mat': 1}

    5if w in counts:6    counts[w] = counts[w] + 17else:
    values this step{'the': 3, 'cat': 1, 'sat': 1, 'on': 1, 'mat': 1} → {'the': 3, 'cat': 2, 'sat': 1, 'on': 1, 'mat': 1}counts

  28. for w in words:

    3counts = {}4for w in words:5    if w in counts:

  29. stdout ← RESULT: {'cat': 2, 'mat': 1, 'on': 1, 'sat': 1, 'the': 3}

    8        counts[w] = 19print('RESULT:', {k: counts[k] for k in sorted(counts)})
    values this stepRESULT: {'cat': 2, 'mat': 1, 'on': 1, 'sat': 1, 'the': 3}stdout

The Pythonic way

Counter(text.split()) tokenises and tallies in one call. The result is a Counter (a dict subclass) with the same counts.

library.py 
from collections import Counter
text = 'the cat sat on the mat the cat'
counts = Counter(text.split())
print('RESULT:', {k: counts[k] for k in sorted(counts)})

RESULT: {'cat': 2, 'mat': 1, 'on': 1, 'sat': 1, 'the': 3}

Implementation notes

  • The mechanism is identical to python-data-basics/frequency-count (ch05). The distinction is the input: ch05 counts a pre-split label list; this lesson first tokenises a raw text string with .split(). Real NLP pipelines add lowercasing and punctuation stripping before counting.
  • .split() with no argument splits on any whitespace run and ignores leading/trailing whitespace — equivalent to .strip().split().
  • Counter.most_common(n) returns the n highest-frequency tokens, useful for finding stop-words or topic keywords.