Detecting and filtering outliers

Filtering or transforming outliers is largely a matter of applying array operations. Consider a DataFrame with some normally distributed data:

import numpy as np
import pandas as pd


df = pd.DataFrame(np.random.randn(1000, 4))

df.describe()

	0	1	2	3
count	1000.000000	1000.000000	1000.000000	1000.000000
mean	0.022943	0.034915	0.018331	0.018969
std	1.046883	0.987310	0.972618	1.009193
min	-3.870700	-2.833648	-3.466318	-3.491794
25%	-0.664047	-0.621664	-0.603689	-0.671775
50%	0.012635	0.026107	0.028248	0.017063
75%	0.736662	0.697112	0.636774	0.743254
max	3.700035	3.006204	2.751574	3.405041

Suppose you want to find values in one of the columns whose absolute value is greater than 3:

col = df[1]

col[col.abs() > 3]

365    3.006204
Name: 1, dtype: float64

To select all rows where value is greater than 3 or less than -3 in one of the columns, you can apply pandas.DataFrame.any to a Boolean DataFrame, using any(axis=1) to check if a value is in a row:

df[(df.abs() > 3).any(axis=1)]

	0	1	2	3
67	-0.065879	1.783196	0.554033	-3.000936
123	0.246540	-0.588655	1.366174	-3.491794
209	-3.615275	-1.539901	-1.109978	1.557272
326	-3.543526	-0.123145	-1.166289	-0.793547
357	1.168551	-0.951635	-0.892777	3.405041
361	3.116807	-0.184181	0.694654	-1.116010
365	-0.274058	3.006204	0.638351	-0.117403
384	-3.006891	0.871370	-0.888511	-0.498219
388	1.104036	0.127207	1.306627	3.164983
504	-0.344477	1.190462	-3.466318	-1.577547
711	3.700035	0.449643	-0.130976	-0.231090
841	-3.870700	0.165213	-0.401433	1.267149
956	3.188822	-0.048598	0.921613	-0.281664
957	-0.326832	-0.324983	0.384806	-3.062165

On this basis, the values can be limited to an interval between -3 and 3. For this we use the instruction np.sign(df), which generates values 1 and -1, depending on whether the values in df are positive or negative:

df[df.abs() > 3] = np.sign(df) * 3

df.describe()

	0	1	2	3
count	1000.000000	1000.000000	1000.000000	1000.000000
mean	0.023974	0.034909	0.018797	0.018954
std	1.037146	0.987292	0.971056	1.005623
min	-3.000000	-2.833648	-3.000000	-3.000000
25%	-0.664047	-0.621664	-0.603689	-0.671775
50%	0.012635	0.026107	0.028248	0.017063
75%	0.736662	0.697112	0.636774	0.743254
max	3.000000	3.000000	2.751574	3.000000