Convert `dtype`#

Sometimes the pandas data types do not fit really well. This can be due to serialisation formats that do not contain type information, for example. However, sometimes you should also change the type to achieve better performance – either more manipulation possibilities or less memory requirements. In the following examples, we will make different conversions of a Series:

[1]:

import numpy as np
import pandas as pd

[2]:

rng = np.random.default_rng()
s = pd.Series(rng.normal(size=7))

[3]:

[3]:

0    0.330852
1    0.703606
2   -0.773463
3   -0.814739
4    1.677859
5    0.992312
6    0.175644
dtype: float64

Automatic conversion#

pandas.Series.convert_dtypes tries to convert a Series to a type that supports NA. In the case of our Series, the type is changed from float64 to Float64:

[4]:

s.convert_dtypes()

[4]:

0    0.330852
1    0.703606
2   -0.773463
3   -0.814739
4    1.677859
5    0.992312
6    0.175644
dtype: Float64

Unfortunately, however, with convert_dtypes I have little control over what data type is converted to. Therefore, I prefer pandas.Series.astype:

[5]:

s.astype("Float32")

[5]:

0    0.330852
1    0.703606
2   -0.773463
3   -0.814739
4    1.677859
5    0.992312
6    0.175644
dtype: Float32

Using the correct type can save memory. The usual data type is 8 bytes wide, for example int64 or float64. If you can use a narrower type, this will significantly reduce memory consumption, allowing you to process more data. You can use NumPy to check the limits of integer and float types:

[6]:

np.iinfo("int64")

[6]:

iinfo(min=-9223372036854775808, max=9223372036854775807, dtype=int64)

[7]:

np.finfo("float32")

[7]:

finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)

[8]:

np.finfo("float64")

[8]:

finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)

Memory usage#

To calculate the memory consumption of the Series, you can use pandas.Series.nbytes to determine the memory used by the data. pandas.Series.memory_usage also records the index memory and the data type. With deep=True you can also determine the memory consumption at system level.

[9]:

s.nbytes

[9]:

[10]:

s.astype("Float32").nbytes

[10]:

[11]:

s.memory_usage()

[11]:

[12]:

s.astype("Float32").memory_usage()

[12]:

[13]:

s.memory_usage(deep=True)

[13]:

String and category types#

The pandas.Series.astype method can also convert numeric series into strings if you pass str. Note the dtype in the following example:

[14]:

s.astype(str)

[14]:

0    0.33085233447486595
1     0.7036061214691522
2    -0.7734631836438829
3    -0.8147390382513203
4     1.6778586038914356
5     0.9923123929031976
6    0.17564372049973478
dtype: object

[15]:

s.astype(str).memory_usage()

[15]:

[16]:

s.astype(str).memory_usage(deep=True)

[16]:

To convert to a categorical type, you can pass 'category' as the type:

[17]:

s.astype(str).astype("category")

[17]:

0    0.33085233447486595
1     0.7036061214691522
2    -0.7734631836438829
3    -0.8147390382513203
4     1.6778586038914356
5     0.9923123929031976
6    0.17564372049973478
dtype: category
Categories (7, object): ['-0.7734631836438829', '-0.8147390382513203', '0.17564372049973478', '0.33085233447486595', '0.7036061214691522', '0.9923123929031976', '1.6778586038914356']

A categorical Series is useful for string data and can lead to large memory savings. This is because when converting to categorical data, pandas no longer uses Python strings for each value, but repeating values are not duplicated. You still have all the features of the str attribute, but you save a lot of memory when you have a lot of duplicate values and you increase performance because you don’t have to do as many string operations.

[18]:

s.astype("category").memory_usage(deep=True)

[18]:

Ordered categories#

To create ordered categories, you need to define your own pandas.CategoricalDtype:

[19]:

from pandas.api.types import CategoricalDtype

sorted = pd.Series(sorted(set(s)))
cat_dtype = CategoricalDtype(categories=sorted, ordered=True)

s.astype(cat_dtype)

[19]:

0    0.330852
1    0.703606
2   -0.773463
3   -0.814739
4    1.677859
5    0.992312
6    0.175644
dtype: category
Categories (7, float64): [-0.814739 < -0.773463 < 0.175644 < 0.330852 < 0.703606 < 0.992312 < 1.677859]

[20]:

s.astype(cat_dtype).memory_usage(deep=True)

[20]:

The following table lists the types you can pass to astype.

Data type	Description
`str`, `'str'`	convert to Python string
`'string'`	convert to Pandas string with `pandas.NA`
`int`, `'int'`, `'int64'`	convert to NumPy `int64`
`'int32'`, `'uint32'`	convert to NumPy `int32`
`'Int64'`	convert to pandas `Int64` with `pandas.NA`
`float`, `'float'`, `'float64'`	convert to floats
`'category'`	convert to `CategoricalDtype` with `pandas.NA`

Conversion to other data types#

The pandas.Series.to_numpy method or the pandas.Series.values property gives us a NumPy array of values, and pandas.Series.to_list returns a Python list of values. Why would you want to do this? pandas objects are usually much more user-friendly and the code is easier to read. Also, python lists will be much slower to process. With pandas.Series.to_frame you can create a DataFrame with a single column, if necessary:

[21]:

s.to_frame()

[21]:

	0
0	0.330852
1	0.703606
2	-0.773463
3	-0.814739
4	1.677859
5	0.992312
6	0.175644

The function pandas.to_datetime can also be useful to convert values in pandas to date and time.

Convert dtype#