Drop Duplicates — Keep First - Data Cleaning Step by Step

Remove duplicate entries, keeping only the first occurrence of each value. By hand, walk the list and collect each name into result only when it has not been seen before. With pandas, DataFrame.drop_duplicates() performs the same operation in one call.

By hand

For each name, check if name not in seen before appending to result. On a new name, append it and record seen[name] = True. Repeated names skip both steps. The trace shows result and seen growing only on first-occurrence iterations.

                            naive.py
                        
Replay: real traced execution (multi-file project)

                            names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']
seen = {}
result = []
for name in names:
    if name not in seen:
        result.append(name)
        seen[name] = True
print('RESULT:', result)

                        
1names ← ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']1names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']2seen = {}⋯values this step['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']names
2seen ← {}1names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']2seen = {}3result = []⋯values this step{}seen
3result ← []⋯2seen = {}3result = []4for name in names:⋯values this step[]result
4name ← 'Alice'⋯3result = []4for name in names:5    if name not in seen:⋯values this step'Alice'name
5if name not in seen:⋯4for name in names:5    if name not in seen:6        result.append(name)⋯
6result ← ['Alice']⋯5if name not in seen:6    result.append(name)7    seen[name] = True⋯values this step[] → ['Alice']result
7seen ← {'Alice': True}⋯6        result.append(name)7        seen[name] = True8print('RESULT:', result)values this step{} → {'Alice': True}seen
8name ← 'Bob'⋯3result = []4for name in names:5    if name not in seen:⋯values this step'Alice' → 'Bob'name
9if name not in seen:⋯4for name in names:5    if name not in seen:6        result.append(name)⋯
10result ← ['Alice', 'Bob']⋯5if name not in seen:6    result.append(name)7    seen[name] = True⋯values this step['Alice'] → ['Alice', 'Bob']result
11seen ← {'Alice': True, 'Bob': True}⋯6        result.append(name)7        seen[name] = True8print('RESULT:', result)values this step{'Alice': True} → {'Alice': True, 'Bob': True}seen
12name ← 'Alice'⋯3result = []4for name in names:5    if name not in seen:⋯values this step'Bob' → 'Alice'name
13if name not in seen:⋯4for name in names:5    if name not in seen:6        result.append(name)⋯
14name ← 'Carol'⋯3result = []4for name in names:5    if name not in seen:⋯values this step'Alice' → 'Carol'name
15if name not in seen:⋯4for name in names:5    if name not in seen:6        result.append(name)⋯
16result ← ['Alice', 'Bob', 'Carol']⋯5if name not in seen:6    result.append(name)7    seen[name] = True⋯values this step['Alice', 'Bob'] → ['Alice', 'Bob', 'Carol']result
17seen ← {'Alice': True, 'Bob': True, 'Carol': True}⋯6        result.append(name)7        seen[name] = True8print('RESULT:', result)values this step{'Alice': True, 'Bob': True} → {'Alice': True, 'Bob': True, 'Carol': True}seen
18name ← 'Bob'⋯3result = []4for name in names:5    if name not in seen:⋯values this step'Carol' → 'Bob'name
19if name not in seen:⋯4for name in names:5    if name not in seen:6        result.append(name)⋯
20name ← 'Dave'⋯3result = []4for name in names:5    if name not in seen:⋯values this step'Bob' → 'Dave'name
21if name not in seen:⋯4for name in names:5    if name not in seen:6        result.append(name)⋯
22result ← ['Alice', 'Bob', 'Carol', 'Dave']⋯5if name not in seen:6    result.append(name)7    seen[name] = True⋯values this step['Alice', 'Bob', 'Carol'] → ['Alice', 'Bob', 'Carol', 'Dave']result
23seen ← {'Alice': True, 'Bob': True, 'Carol': True, 'Dave': True}⋯6        result.append(name)7        seen[name] = True8print('RESULT:', result)values this step{'Alice': True, 'Bob': True, 'Carol': True} → {'Alice': True, 'Bob': True, 'Carol': True, 'Dave': True}seen
24for name in names:⋯3result = []4for name in names:5    if name not in seen:⋯
25stdout ← RESULT: ['Alice', 'Bob', 'Carol', 'Dave']⋯7        seen[name] = True8print('RESULT:', result)values this stepRESULT: ['Alice', 'Bob', 'Carol', 'Dave']stdout

With pandas

df.drop_duplicates() returns a new DataFrame with duplicate rows removed, keeping the first occurrence. The snapshot shows the shape before and after so the number of dropped rows is immediately visible.

                            library.py
                        

                            import pandas as pd
from dalib.display import set_display
set_display()

names = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Dave']
df = pd.DataFrame({'name': names})
clean = df.drop_duplicates()
result = clean['name'].tolist()
print('columns:', df.columns.tolist())
print('shape before:', df.shape)
print('shape after:', clean.shape)
print('RESULT:', result)

                        

columns: ['name']
shape before: (6, 1)
shape after: (4, 1)
RESULT: ['Alice', 'Bob', 'Carol', 'Dave']

Implementation notes

drop_duplicates() uses keep='first' by default. Pass keep='last' to retain the last occurrence, or keep=False to drop every row that has any duplicate (keeping nothing).
The returned DataFrame preserves the original index labels. Use .reset_index(drop=True) if a clean 0-based index is needed.
Pass subset=['col'] to deduplicate on a specific column while keeping all other columns.
Cross-reference: find-exact-duplicates (this chapter) to inspect which rows would be dropped before committing to the removal.