Subdividing and categorising data

Continuous data is often divided into domains or otherwise grouped for analysis.

Suppose you have data on a group of people in a study that you want to divide into discrete age groups. For this, we generate a dataframe with 250 entries between 0 and 99:

import numpy as np
import pandas as pd


ages = np.random.randint(0, 99, 250)
df = pd.DataFrame({"Age": ages})

df

	Age
0	36
1	20
2	54
3	63
4	60
...	...
245	35
246	93
247	97
248	84
249	9

250 rows × 1 columns

Afterwards, pandas offers us a simple way to divide the results into ten ranges with pandas.cut. To get only whole years, we additionally set precision=0:

cats = pd.cut(ages, 10, precision=0)

cats

[(29.0, 39.0], (20.0, 29.0], (49.0, 59.0], (59.0, 69.0], (59.0, 69.0], ..., (29.0, 39.0], (88.0, 98.0], (88.0, 98.0], (78.0, 88.0], (-0.1, 10.0]]
Length: 250
Categories (10, interval[float64, right]): [(-0.1, 10.0] < (10.0, 20.0] < (20.0, 29.0] < (29.0, 39.0] ... (59.0, 69.0] < (69.0, 78.0] < (78.0, 88.0] < (88.0, 98.0]]

With pandas.Categorical.categories you can display the categories:

cats.categories

IntervalIndex([(-0.1, 10.0], (10.0, 20.0], (20.0, 29.0], (29.0, 39.0],
               (39.0, 49.0], (49.0, 59.0], (59.0, 69.0], (69.0, 78.0],
               (78.0, 88.0], (88.0, 98.0]],
              dtype='interval[float64, right]')

… or even just a single category:

cats.categories[0]

Interval(-0.1, 10.0, closed='right')

With pandas.Categorical.codes you can display an array where for each value the corresponding category is shown:

cats.codes

array([3, 2, 5, 6, 6, 4, 6, 9, 5, 9, 9, 1, 2, 8, 7, 3, 9, 4, 4, 7, 1, 4,
       0, 8, 6, 6, 0, 2, 1, 4, 9, 0, 6, 5, 1, 4, 8, 0, 3, 1, 0, 9, 4, 2,
       5, 8, 3, 8, 3, 2, 3, 9, 8, 2, 2, 8, 5, 0, 8, 9, 0, 8, 1, 5, 8, 9,
       3, 6, 4, 8, 2, 4, 3, 9, 5, 9, 8, 1, 9, 7, 4, 1, 0, 9, 2, 0, 0, 9,
       0, 5, 6, 8, 2, 9, 1, 6, 8, 6, 0, 8, 2, 5, 5, 9, 5, 4, 1, 7, 0, 3,
       6, 8, 0, 7, 6, 2, 0, 3, 4, 6, 5, 9, 6, 2, 0, 4, 3, 7, 7, 0, 7, 1,
       9, 9, 3, 0, 9, 8, 9, 7, 1, 7, 6, 3, 2, 8, 6, 2, 9, 9, 3, 7, 6, 7,
       3, 3, 0, 9, 1, 5, 3, 6, 4, 6, 2, 6, 4, 9, 2, 7, 1, 7, 6, 4, 1, 5,
       2, 1, 5, 4, 9, 4, 7, 0, 3, 8, 7, 6, 7, 6, 7, 7, 2, 2, 7, 3, 0, 9,
       3, 7, 6, 3, 6, 9, 1, 2, 3, 7, 7, 7, 8, 5, 6, 0, 6, 1, 1, 6, 0, 5,
       2, 5, 1, 9, 1, 0, 6, 9, 4, 5, 9, 6, 1, 8, 5, 6, 9, 6, 8, 7, 9, 1,
       2, 4, 0, 3, 9, 9, 8, 0], dtype=int8)

With value_counts we can now look at how the number is distributed among the individual areas:

pd.Series(cats).value_counts()

(88.0, 98.0]    35
(59.0, 69.0]    32
(-0.1, 10.0]    26
(69.0, 78.0]    25
(10.0, 20.0]    23
(20.0, 29.0]    23
(29.0, 39.0]    23
(78.0, 88.0]    23
(39.0, 49.0]    20
(49.0, 59.0]    20
Name: count, dtype: int64

It is striking that the age ranges do not contain an equal number of years, but with 20.0, 29.0 and 69.0, 78.0 two ranges contain only 9 years. This is due to the fact that the age range only extends from 0 to 98:

df.min()

Age    0
dtype: int64

df.max()

Age    98
dtype: int64

With pandas.qcut, on the other hand, the set is divided into areas that are approximately the same size:

cats = pd.qcut(ages, 10, precision=0)

pd.Series(cats).value_counts()

(33.0, 41.0]    27
(54.0, 63.0]    27
(-1.0, 9.0]     26
(9.0, 20.0]     26
(82.0, 91.0]    26
(71.0, 82.0]    25
(91.0, 98.0]    24
(20.0, 33.0]    23
(41.0, 54.0]    23
(63.0, 71.0]    23
Name: count, dtype: int64

If we want to ensure that each age group actually includes exactly ten years, we can specify this directly with pandas.Categorical:

age_groups = ["{0} - {1}".format(i, i + 9) for i in range(0, 99, 10)]
cats = pd.Categorical(age_groups)

cats.categories

Index(['0 - 9', '10 - 19', '20 - 29', '30 - 39', '40 - 49', '50 - 59',
       '60 - 69', '70 - 79', '80 - 89', '90 - 99'],
      dtype='object')

For grouping we can now use pandas.cut. However, the number of labels must be one less than the number of edges:

df["Age group"] = pd.cut(df.Age, range(0, 101, 10), right=False, labels=cats)

df

	Age	Age group
0	36	30 - 39
1	20	20 - 29
2	54	50 - 59
3	63	60 - 69
4	60	60 - 69
...	...	...
245	35	30 - 39
246	93	90 - 99
247	97	90 - 99
248	84	80 - 89
249	9	0 - 9

250 rows × 2 columns