Selecting and Filtering

Filter with Multiple Conditions

Combine two per-row boolean tests with Python's and in a loop to build a single mask, then keep matching rows. With pandas, element-wise & combines two boolean Series before indexing the DataFrame — and and or are not valid for Series and will raise an error.

By hand

With pandas

(df['age'] > lo) & (df['score'] < hi) combines two boolean Series elementwise. Parentheses around each condition are required because & has higher operator precedence than > and <.

naive.py 
# trace: ignore records
records = [
    {'name': 'al', 'age': 20, 'score': 55},
    {'name': 'bo', 'age': 25, 'score': 82},
    {'name': 'cy', 'age': 22, 'score': 71},
    {'name': 'di', 'age': 28, 'score': 90},
]
lo, hi = 21, 88
mask = []
for r in records:
    mask.append(r['age'] > lo and r['score'] < hi)
names = []
for i in range(len(records)):
    if mask[i]:
        names.append(records[i]['name'])
print('RESULT:', names)
library.py 
import pandas as pd
from dalib.display import set_display
set_display()

records = [
    {'name': 'al', 'age': 20, 'score': 55},
    {'name': 'bo', 'age': 25, 'score': 82},
    {'name': 'cy', 'age': 22, 'score': 71},
    {'name': 'di', 'age': 28, 'score': 90},
]
lo, hi = 21, 88
df = pd.DataFrame(records)
mask = (df['age'] > lo) & (df['score'] < hi)
filtered = df[mask]
print('mask:', mask.tolist())
print('names:', filtered['name'].tolist())
print('shape:', filtered.shape)
print('RESULT:', filtered['name'].tolist())

mask: [False, True, True, False]
names: ['bo', 'cy']
shape: (2, 3)
RESULT: ['bo', 'cy']

Implementation notes

  • Use & (bitwise AND) and | (bitwise OR) to combine boolean Series — NOT Python's and / or. and/or call bool() on the whole Series, which raises ValueError: The truth value of a Series is ambiguous.
  • Parentheses around each condition are mandatory: (df['a'] > lo) & (df['b'] < hi). Without them, & binds tighter than > and the expression parses incorrectly.
  • This is the same operator rule as NumPy: (a > lo) & (a < hi) — see combine-two-masks (python-numpy ch06).
  • For more than two conditions, chain & or |: (c1) & (c2) & (c3). Alternatively, df.query("age > 21 and score < 88") allows Python-style and/or syntax inside a string.