Duplicates

Find Exact Duplicates

Flag each element that is a repeat of an earlier occurrence. By hand, keep a seen dict and mark each element True if it is already present. With pandas, DataFrame.duplicated() returns a boolean Series where True marks every row after its first occurrence.

By hand

Before appending to flags, check whether name is already in seen. Then record seen[name] = True so later occurrences are caught. The trace shows seen growing as new names are encountered and flags accumulating a True each time a repeat is detected.

naive.py
Replay: real traced execution (multi-file project)
names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']
seen = {}
flags = []
for name in names:
    flags.append(name in seen)
    seen[name] = True
print('RESULT:', flags)

  1. names ← ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']

    1names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']2seen = {}
    values this step['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']names

  2. seen ← {}

    1names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']2seen = {}3flags = []
    values this step{}seen

  3. flags ← []

    2seen = {}3flags = []4for name in names:
    values this step[]flags

  4. name ← 'Alice'

    3flags = []4for name in names:5    flags.append(name in seen)
    values this step'Alice'name

  5. flags ← [False]

    4for name in names:5    flags.append(name in seen)6    seen[name] = True
    values this step[] → [False]flags

  6. seen ← {'Alice': True}

    5    flags.append(name in seen)6    seen[name] = True7print('RESULT:', flags)
    values this step{} → {'Alice': True}seen

  7. name ← 'Bob'

    3flags = []4for name in names:5    flags.append(name in seen)
    values this step'Alice' → 'Bob'name

  8. flags ← [False, False]

    4for name in names:5    flags.append(name in seen)6    seen[name] = True
    values this step[False] → [False, False]flags

  9. seen ← {'Alice': True, 'Bob': True}

    5    flags.append(name in seen)6    seen[name] = True7print('RESULT:', flags)
    values this step{'Alice': True} → {'Alice': True, 'Bob': True}seen

  10. name ← 'Alice'

    3flags = []4for name in names:5    flags.append(name in seen)
    values this step'Bob' → 'Alice'name

  11. flags ← [False, False, True]

    4for name in names:5    flags.append(name in seen)6    seen[name] = True
    values this step[False, False] → [False, False, True]flags

  12. seen[name] = True

    5    flags.append(name in seen)6    seen[name] = True7print('RESULT:', flags)

  13. name ← 'Carol'

    3flags = []4for name in names:5    flags.append(name in seen)
    values this step'Alice' → 'Carol'name

  14. flags ← [False, False, True, False]

    4for name in names:5    flags.append(name in seen)6    seen[name] = True
    values this step[False, False, True] → [False, False, True, False]flags

  15. seen ← {'Alice': True, 'Bob': True, 'Carol': True}

    5    flags.append(name in seen)6    seen[name] = True7print('RESULT:', flags)
    values this step{'Alice': True, 'Bob': True} → {'Alice': True, 'Bob': True, 'Carol': True}seen

  16. name ← 'Bob'

    3flags = []4for name in names:5    flags.append(name in seen)
    values this step'Carol' → 'Bob'name

  17. flags ← [False, False, True, False, True]

    4for name in names:5    flags.append(name in seen)6    seen[name] = True
    values this step[False, False, True, False] → [False, False, True, False, True]flags

  18. seen[name] = True

    5    flags.append(name in seen)6    seen[name] = True7print('RESULT:', flags)

  19. name ← 'Dave'

    3flags = []4for name in names:5    flags.append(name in seen)
    values this step'Bob' → 'Dave'name

  20. flags ← [False, False, True, False, True, False]

    4for name in names:5    flags.append(name in seen)6    seen[name] = True
    values this step[False, False, True, False, True] → [False, False, True, False, True, False]flags

  21. seen ← {'Alice': True, 'Bob': True, 'Carol': True, 'Dave': True}

    5    flags.append(name in seen)6    seen[name] = True7print('RESULT:', flags)
    values this step{'Alice': True, 'Bob': True, 'Carol': True} → {'Alice': True, 'Bob': True, 'Carol': True, 'Dave': True}seen

  22. for name in names:

    3flags = []4for name in names:5    flags.append(name in seen)

  23. stdout ← RESULT: [False, False, True, False, True, False]

    6    seen[name] = True7print('RESULT:', flags)
    values this stepRESULT: [False, False, True, False, True, False]stdout

With pandas

df.duplicated() scans rows and marks every row after the first occurrence of each value as True. The result is a boolean dtype: bool Series.

library.py 
import pandas as pd
from dalib.display import set_display
set_display()

names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']
df = pd.DataFrame({'name': names})
flags = df.duplicated()
result = flags.tolist()
print('index:', flags.index.tolist())
print('dtype:', flags.dtype)
print('values:', flags.tolist())
print('RESULT:', result)

index: [0, 1, 2, 3, 4, 5]
dtype: bool
values: [False, False, True, False, True, False]
RESULT: [False, False, True, False, True, False]

Implementation notes

  • duplicated() uses keep='first' by default — the first occurrence is False (not a duplicate) and all later ones are True. Use keep='last' to keep the last occurrence instead, or keep=False to flag every copy including the first.
  • Pass subset=['col1', 'col2'] to check duplicates only on specific columns rather than all columns.
  • Cross-reference: drop-duplicates-keep-first (this chapter) to remove the flagged rows in one step rather than just identifying them.