pbPassingBI
/
Wrangling & cleaning beginner 6 min

Text operations

The .str accessor, and the cleaning steps worth doing on every import.

What you'll be able to do
  • Clean text columns with .str
  • Extract and replace substrings
  • Split a column into several

The .str accessor

df['name'].str.lower()
df['name'].str.upper()
df['name'].str.strip()          # whitespace at both ends
df['name'].str.len()
df['name'].str.replace('Ltd', 'Limited')
df['name'].str.contains('Corp', na=False)
df['name'].str.startswith('A')

Everything text-related goes through .str. Without it you are calling the method on the Series object rather than on its values.

The standard clean

df['email'] = df['email'].str.strip().str.lower()

Run this on every text key before joining or deduplicating. Trailing whitespace and mixed case are the two reasons merges silently fail to match on values that look identical.

Splitting

df[['first', 'last']] = df['full_name'].str.split(' ', n=1, expand=True)
df['domain'] = df['email'].str.split('@').str[1]

expand=True returns a DataFrame you can assign to several columns. n=1 splits only on the first occurrence, so a middle name does not break the assignment.

Extracting

df['digits'] = df['phone'].str.replace(r'\D', '', regex=True)
df['code']   = df['sku'].str.extract(r'^([A-Z]{3})')
df['num']    = df['label'].str.extract(r'(\d+)').astype(float)

extract takes a regular expression with a capture group and returns what it captured, or NaN where there was no match.

na=False

contains() on a column with nulls

df[df['name'].str.contains('Ltd')] raises an error when the column has any NaN. Pass na=False to treat missing as not-matching.

Key points
  • All text methods go through the .str accessor
  • strip().lower() on keys prevents silent merge failures
  • str.contains needs na=False when the column has nulls
Check yourself