{ "cells": [ { "cell_type": "markdown", "id": "73bd3476", "metadata": {}, "source": [ "# Convert `dtype`\n", "\n", "Sometimes the pandas data types do not fit really well. This can be due to serialisation formats that do not contain type information, for example. However, sometimes you should also change the type to achieve better performance – either more manipulation possibilities or less memory requirements. In the following examples, we will make different conversions of a `Series`:" ] }, { "cell_type": "code", "execution_count": 1, "id": "69abe2d3", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:31.791981Z", "iopub.status.busy": "2026-05-16T14:17:31.791817Z", "iopub.status.idle": "2026-05-16T14:17:32.142849Z", "shell.execute_reply": "2026-05-16T14:17:32.142551Z", "shell.execute_reply.started": "2026-05-16T14:17:31.791963Z" } }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "id": "c2a35bac", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:32.143343Z", "iopub.status.busy": "2026-05-16T14:17:32.143181Z", "iopub.status.idle": "2026-05-16T14:17:32.145742Z", "shell.execute_reply": "2026-05-16T14:17:32.145440Z", "shell.execute_reply.started": "2026-05-16T14:17:32.143328Z" } }, "outputs": [], "source": [ "rng = np.random.default_rng()\n", "s = pd.Series(rng.normal(size=7))" ] }, { "cell_type": "code", "execution_count": 3, "id": "98ffcd09", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:32.146360Z", "iopub.status.busy": "2026-05-16T14:17:32.146256Z", "iopub.status.idle": "2026-05-16T14:17:32.151148Z", "shell.execute_reply": "2026-05-16T14:17:32.150812Z", "shell.execute_reply.started": "2026-05-16T14:17:32.146352Z" }, "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "0 0.247177\n", "1 -0.541495\n", "2 0.627878\n", "3 -0.160339\n", "4 0.798170\n", "5 -0.075034\n", "6 -0.371027\n", "dtype: float64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s" ] }, { "cell_type": "markdown", "id": "ab32810f", "metadata": {}, "source": [ "## Automatic conversion\n", "\n", "[pandas.Series.convert_dtypes](https://pandas.pydata.org/docs/reference/api/pandas.Series.convert_dtypes.html) tries to convert a `Series` to a type that supports `NA`. In the case of our `Series`, the type is changed from `float64` to `Float64`:" ] }, { "cell_type": "code", "execution_count": 4, "id": "c9738255", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:32.152090Z", "iopub.status.busy": "2026-05-16T14:17:32.151728Z", "iopub.status.idle": "2026-05-16T14:17:32.154967Z", "shell.execute_reply": "2026-05-16T14:17:32.154677Z", "shell.execute_reply.started": "2026-05-16T14:17:32.152067Z" } }, "outputs": [ { "data": { "text/plain": [ "0 0.247177\n", "1 -0.541495\n", "2 0.627878\n", "3 -0.160339\n", "4 0.79817\n", "5 -0.075034\n", "6 -0.371027\n", "dtype: Float64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.convert_dtypes()" ] }, { "cell_type": "markdown", "id": "70ef14c8", "metadata": {}, "source": [ "Unfortunately, however, with `convert_dtypes` I have little control over what data type is converted to. Therefore, I prefer [pandas.Series.astype](https://pandas.pydata.org/docs/reference/api/pandas.Series.astype.html):" ] }, { "cell_type": "code", "execution_count": 5, "id": "4c6de2eb", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:32.155447Z", "iopub.status.busy": "2026-05-16T14:17:32.155334Z", "iopub.status.idle": "2026-05-16T14:17:32.158599Z", "shell.execute_reply": "2026-05-16T14:17:32.158350Z", "shell.execute_reply.started": "2026-05-16T14:17:32.155438Z" } }, "outputs": [ { "data": { "text/plain": [ "0 0.247177\n", "1 -0.541495\n", "2 0.627878\n", "3 -0.160339\n", "4 0.79817\n", "5 -0.075034\n", "6 -0.371027\n", "dtype: Float32" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(\"Float32\")" ] }, { "cell_type": "markdown", "id": "43cc3259-4e93-4b05-a826-24db437bdecd", "metadata": {}, "source": [ "However, if non-convertible values are included, an error will be returned:" ] }, { "cell_type": "code", "execution_count": 6, "id": "29f5c8d4-de3e-420b-ae01-1b2c3d6abb5b", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:32.161135Z", "iopub.status.busy": "2026-05-16T14:17:32.161003Z", "iopub.status.idle": "2026-05-16T14:17:32.164192Z", "shell.execute_reply": "2026-05-16T14:17:32.163830Z", "shell.execute_reply.started": "2026-05-16T14:17:32.161124Z" } }, "outputs": [ { "data": { "text/plain": [ "0 65.0\n", "1 NaN\n", "2 40.0\n", "dtype: float64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rng = np.random.default_rng()\n", "n = pd.Series([rng.integers(127), np.nan, rng.integers(127)])\n", "n" ] }, { "cell_type": "code", "execution_count": 7, "id": "72500da6-7057-4298-a18a-704b9d0e8eea", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:32.164841Z", "iopub.status.busy": "2026-05-16T14:17:32.164626Z", "iopub.status.idle": "2026-05-16T14:17:32.467206Z", "shell.execute_reply": "2026-05-16T14:17:32.466230Z", "shell.execute_reply.started": "2026-05-16T14:17:32.164831Z" } }, "outputs": [ { "ename": "IntCastingNaNError", "evalue": "Cannot convert non-finite values (NA or inf) to integer", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mIntCastingNaNError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[7], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mn\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mastype\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mint8\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/generic.py:6643\u001b[0m, in \u001b[0;36mNDFrame.astype\u001b[0;34m(self, dtype, copy, errors)\u001b[0m\n\u001b[1;32m 6637\u001b[0m results \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m 6638\u001b[0m ser\u001b[38;5;241m.\u001b[39mastype(dtype, copy\u001b[38;5;241m=\u001b[39mcopy, errors\u001b[38;5;241m=\u001b[39merrors) \u001b[38;5;28;01mfor\u001b[39;00m _, ser \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mitems()\n\u001b[1;32m 6639\u001b[0m ]\n\u001b[1;32m 6641\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 6642\u001b[0m \u001b[38;5;66;03m# else, only a single dtype is given\u001b[39;00m\n\u001b[0;32m-> 6643\u001b[0m new_data \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_mgr\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mastype\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdtype\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcopy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcopy\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 6644\u001b[0m res \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_constructor_from_mgr(new_data, axes\u001b[38;5;241m=\u001b[39mnew_data\u001b[38;5;241m.\u001b[39maxes)\n\u001b[1;32m 6645\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m res\u001b[38;5;241m.\u001b[39m__finalize__(\u001b[38;5;28mself\u001b[39m, method\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mastype\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/internals/managers.py:430\u001b[0m, in \u001b[0;36mBaseBlockManager.astype\u001b[0;34m(self, dtype, copy, errors)\u001b[0m\n\u001b[1;32m 427\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m using_copy_on_write():\n\u001b[1;32m 428\u001b[0m copy \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[0;32m--> 430\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mapply\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 431\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mastype\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 432\u001b[0m \u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 433\u001b[0m \u001b[43m \u001b[49m\u001b[43mcopy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcopy\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 434\u001b[0m \u001b[43m \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 435\u001b[0m \u001b[43m \u001b[49m\u001b[43musing_cow\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43musing_copy_on_write\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 436\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/internals/managers.py:363\u001b[0m, in \u001b[0;36mBaseBlockManager.apply\u001b[0;34m(self, f, align_keys, **kwargs)\u001b[0m\n\u001b[1;32m 361\u001b[0m applied \u001b[38;5;241m=\u001b[39m b\u001b[38;5;241m.\u001b[39mapply(f, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m 362\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 363\u001b[0m applied \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mgetattr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mb\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mf\u001b[49m\u001b[43m)\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 364\u001b[0m result_blocks \u001b[38;5;241m=\u001b[39m extend_blocks(applied, result_blocks)\n\u001b[1;32m 366\u001b[0m out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mtype\u001b[39m(\u001b[38;5;28mself\u001b[39m)\u001b[38;5;241m.\u001b[39mfrom_blocks(result_blocks, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39maxes)\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/internals/blocks.py:758\u001b[0m, in \u001b[0;36mBlock.astype\u001b[0;34m(self, dtype, copy, errors, using_cow, squeeze)\u001b[0m\n\u001b[1;32m 755\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCan not squeeze with more than one column.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 756\u001b[0m values \u001b[38;5;241m=\u001b[39m values[\u001b[38;5;241m0\u001b[39m, :] \u001b[38;5;66;03m# type: ignore[call-overload]\u001b[39;00m\n\u001b[0;32m--> 758\u001b[0m new_values \u001b[38;5;241m=\u001b[39m \u001b[43mastype_array_safe\u001b[49m\u001b[43m(\u001b[49m\u001b[43mvalues\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcopy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcopy\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 760\u001b[0m new_values \u001b[38;5;241m=\u001b[39m maybe_coerce_values(new_values)\n\u001b[1;32m 762\u001b[0m refs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/dtypes/astype.py:237\u001b[0m, in \u001b[0;36mastype_array_safe\u001b[0;34m(values, dtype, copy, errors)\u001b[0m\n\u001b[1;32m 234\u001b[0m dtype \u001b[38;5;241m=\u001b[39m dtype\u001b[38;5;241m.\u001b[39mnumpy_dtype\n\u001b[1;32m 236\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 237\u001b[0m new_values \u001b[38;5;241m=\u001b[39m \u001b[43mastype_array\u001b[49m\u001b[43m(\u001b[49m\u001b[43mvalues\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcopy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcopy\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 238\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\u001b[38;5;167;01mValueError\u001b[39;00m, \u001b[38;5;167;01mTypeError\u001b[39;00m):\n\u001b[1;32m 239\u001b[0m \u001b[38;5;66;03m# e.g. _astype_nansafe can fail on object-dtype of strings\u001b[39;00m\n\u001b[1;32m 240\u001b[0m \u001b[38;5;66;03m# trying to convert to float\u001b[39;00m\n\u001b[1;32m 241\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m errors \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mignore\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/dtypes/astype.py:182\u001b[0m, in \u001b[0;36mastype_array\u001b[0;34m(values, dtype, copy)\u001b[0m\n\u001b[1;32m 179\u001b[0m values \u001b[38;5;241m=\u001b[39m values\u001b[38;5;241m.\u001b[39mastype(dtype, copy\u001b[38;5;241m=\u001b[39mcopy)\n\u001b[1;32m 181\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 182\u001b[0m values \u001b[38;5;241m=\u001b[39m \u001b[43m_astype_nansafe\u001b[49m\u001b[43m(\u001b[49m\u001b[43mvalues\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcopy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcopy\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# in pandas we don't store numpy str dtypes, so convert to object\u001b[39;00m\n\u001b[1;32m 185\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(dtype, np\u001b[38;5;241m.\u001b[39mdtype) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28missubclass\u001b[39m(values\u001b[38;5;241m.\u001b[39mdtype\u001b[38;5;241m.\u001b[39mtype, \u001b[38;5;28mstr\u001b[39m):\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/dtypes/astype.py:101\u001b[0m, in \u001b[0;36m_astype_nansafe\u001b[0;34m(arr, dtype, copy, skipna)\u001b[0m\n\u001b[1;32m 96\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m lib\u001b[38;5;241m.\u001b[39mensure_string_array(\n\u001b[1;32m 97\u001b[0m arr, skipna\u001b[38;5;241m=\u001b[39mskipna, convert_na_value\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m 98\u001b[0m )\u001b[38;5;241m.\u001b[39mreshape(shape)\n\u001b[1;32m 100\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m np\u001b[38;5;241m.\u001b[39missubdtype(arr\u001b[38;5;241m.\u001b[39mdtype, np\u001b[38;5;241m.\u001b[39mfloating) \u001b[38;5;129;01mand\u001b[39;00m dtype\u001b[38;5;241m.\u001b[39mkind \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124miu\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[0;32m--> 101\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43m_astype_float_to_int_nansafe\u001b[49m\u001b[43m(\u001b[49m\u001b[43marr\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcopy\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 103\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m arr\u001b[38;5;241m.\u001b[39mdtype \u001b[38;5;241m==\u001b[39m \u001b[38;5;28mobject\u001b[39m:\n\u001b[1;32m 104\u001b[0m \u001b[38;5;66;03m# if we have a datetime/timedelta array of objects\u001b[39;00m\n\u001b[1;32m 105\u001b[0m \u001b[38;5;66;03m# then coerce to datetime64[ns] and use DatetimeArray.astype\u001b[39;00m\n\u001b[1;32m 107\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m lib\u001b[38;5;241m.\u001b[39mis_np_dtype(dtype, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mM\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n", "File \u001b[0;32m~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/pandas/core/dtypes/astype.py:145\u001b[0m, in \u001b[0;36m_astype_float_to_int_nansafe\u001b[0;34m(values, dtype, copy)\u001b[0m\n\u001b[1;32m 141\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 142\u001b[0m \u001b[38;5;124;03mastype with a check preventing converting NaN to an meaningless integer value.\u001b[39;00m\n\u001b[1;32m 143\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 144\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m np\u001b[38;5;241m.\u001b[39misfinite(values)\u001b[38;5;241m.\u001b[39mall():\n\u001b[0;32m--> 145\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m IntCastingNaNError(\n\u001b[1;32m 146\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot convert non-finite values (NA or inf) to integer\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 147\u001b[0m )\n\u001b[1;32m 148\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m dtype\u001b[38;5;241m.\u001b[39mkind \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mu\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[1;32m 149\u001b[0m \u001b[38;5;66;03m# GH#45151\u001b[39;00m\n\u001b[1;32m 150\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (values \u001b[38;5;241m>\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0\u001b[39m)\u001b[38;5;241m.\u001b[39mall():\n", "\u001b[0;31mIntCastingNaNError\u001b[0m: Cannot convert non-finite values (NA or inf) to integer" ] } ], "source": [ "n.astype(\"int8\")" ] }, { "cell_type": "markdown", "id": "62c1c73f-c268-4a95-9691-08c4f21ae43e", "metadata": {}, "source": [ "Errors such as IntCastingNaNError can be avoided by retaining the original data type using `errors = \"ignore\"` if necessary:" ] }, { "cell_type": "code", "execution_count": 8, "id": "8efe298d-25bb-43ea-8641-f200148826b3", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:38.201676Z", "iopub.status.busy": "2026-05-16T14:17:38.201385Z", "iopub.status.idle": "2026-05-16T14:17:38.206241Z", "shell.execute_reply": "2026-05-16T14:17:38.205824Z", "shell.execute_reply.started": "2026-05-16T14:17:38.201658Z" } }, "outputs": [ { "data": { "text/plain": [ "0 65.0\n", "1 NaN\n", "2 40.0\n", "dtype: float64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "n.astype(\"int8\", errors=\"ignore\")" ] }, { "cell_type": "markdown", "id": "b725610a", "metadata": {}, "source": [ "Using the correct type can save memory. The usual data type is 8 bytes wide, for example `int64` or `float64`. If you can use a narrower type, this will significantly reduce memory consumption, allowing you to process more data. You can use NumPy to check the limits of integer and float types:" ] }, { "cell_type": "code", "execution_count": 9, "id": "46c513b8-06b2-42e4-bd5b-39ca17832003", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:39.173396Z", "iopub.status.busy": "2026-05-16T14:17:39.173113Z", "iopub.status.idle": "2026-05-16T14:17:39.177039Z", "shell.execute_reply": "2026-05-16T14:17:39.176536Z", "shell.execute_reply.started": "2026-05-16T14:17:39.173377Z" } }, "outputs": [ { "data": { "text/plain": [ "iinfo(min=-128, max=127, dtype=int8)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.iinfo(\"int8\")" ] }, { "cell_type": "code", "execution_count": 10, "id": "f9e051ef", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:39.534356Z", "iopub.status.busy": "2026-05-16T14:17:39.534000Z", "iopub.status.idle": "2026-05-16T14:17:39.538113Z", "shell.execute_reply": "2026-05-16T14:17:39.537619Z", "shell.execute_reply.started": "2026-05-16T14:17:39.534334Z" } }, "outputs": [ { "data": { "text/plain": [ "iinfo(min=-9223372036854775808, max=9223372036854775807, dtype=int64)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.iinfo(\"int64\")" ] }, { "cell_type": "code", "execution_count": 11, "id": "add9af88", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:39.930385Z", "iopub.status.busy": "2026-05-16T14:17:39.930098Z", "iopub.status.idle": "2026-05-16T14:17:39.933844Z", "shell.execute_reply": "2026-05-16T14:17:39.933306Z", "shell.execute_reply.started": "2026-05-16T14:17:39.930367Z" } }, "outputs": [ { "data": { "text/plain": [ "finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.finfo(\"float32\")" ] }, { "cell_type": "code", "execution_count": 12, "id": "1a5e1e44", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:40.296176Z", "iopub.status.busy": "2026-05-16T14:17:40.295894Z", "iopub.status.idle": "2026-05-16T14:17:40.299460Z", "shell.execute_reply": "2026-05-16T14:17:40.298952Z", "shell.execute_reply.started": "2026-05-16T14:17:40.296159Z" } }, "outputs": [ { "data": { "text/plain": [ "finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.finfo(\"float64\")" ] }, { "cell_type": "markdown", "id": "ea015364", "metadata": {}, "source": [ "## Memory usage\n", "\n", "To calculate the memory consumption of the `Series`, you can use [pandas.Series.nbytes](https://pandas.pydata.org/docs/reference/api/pandas.Series.nbytes.html) to determine the memory used by the data. [pandas.Series.memory_usage](https://pandas.pydata.org/docs/reference/api/pandas.Series.memory_usage.html) also records the index memory and the data type. With `deep=True` you can also determine the memory consumption at system level." ] }, { "cell_type": "code", "execution_count": 13, "id": "e17aa411", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:41.220601Z", "iopub.status.busy": "2026-05-16T14:17:41.220022Z", "iopub.status.idle": "2026-05-16T14:17:41.224294Z", "shell.execute_reply": "2026-05-16T14:17:41.223878Z", "shell.execute_reply.started": "2026-05-16T14:17:41.220558Z" } }, "outputs": [ { "data": { "text/plain": [ "56" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.nbytes" ] }, { "cell_type": "code", "execution_count": 14, "id": "2a73df20", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:41.610843Z", "iopub.status.busy": "2026-05-16T14:17:41.610281Z", "iopub.status.idle": "2026-05-16T14:17:41.614875Z", "shell.execute_reply": "2026-05-16T14:17:41.614476Z", "shell.execute_reply.started": "2026-05-16T14:17:41.610801Z" } }, "outputs": [ { "data": { "text/plain": [ "35" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(\"Float32\").nbytes" ] }, { "cell_type": "code", "execution_count": 15, "id": "3b3c53d5", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:42.110510Z", "iopub.status.busy": "2026-05-16T14:17:42.110215Z", "iopub.status.idle": "2026-05-16T14:17:42.114023Z", "shell.execute_reply": "2026-05-16T14:17:42.113570Z", "shell.execute_reply.started": "2026-05-16T14:17:42.110491Z" } }, "outputs": [ { "data": { "text/plain": [ "188" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.memory_usage()" ] }, { "cell_type": "code", "execution_count": 16, "id": "f1add8c4", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:42.951863Z", "iopub.status.busy": "2026-05-16T14:17:42.951568Z", "iopub.status.idle": "2026-05-16T14:17:42.956756Z", "shell.execute_reply": "2026-05-16T14:17:42.955472Z", "shell.execute_reply.started": "2026-05-16T14:17:42.951845Z" } }, "outputs": [ { "data": { "text/plain": [ "167" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(\"Float32\").memory_usage()" ] }, { "cell_type": "code", "execution_count": 17, "id": "91108fce", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:43.370306Z", "iopub.status.busy": "2026-05-16T14:17:43.370006Z", "iopub.status.idle": "2026-05-16T14:17:43.373619Z", "shell.execute_reply": "2026-05-16T14:17:43.373167Z", "shell.execute_reply.started": "2026-05-16T14:17:43.370287Z" } }, "outputs": [ { "data": { "text/plain": [ "188" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.memory_usage(deep=True)" ] }, { "cell_type": "markdown", "id": "7859b4e2", "metadata": {}, "source": [ "## String and category types\n", "\n", "The [pandas.Series.astype](https://pandas.pydata.org/docs/reference/api/pandas.Series.astype.html) method can also convert numeric series into strings if you pass `str`. Note the `dtype` in the following example:" ] }, { "cell_type": "code", "execution_count": 18, "id": "98fff9a8", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:44.389987Z", "iopub.status.busy": "2026-05-16T14:17:44.389437Z", "iopub.status.idle": "2026-05-16T14:17:44.396777Z", "shell.execute_reply": "2026-05-16T14:17:44.396315Z", "shell.execute_reply.started": "2026-05-16T14:17:44.389946Z" } }, "outputs": [ { "data": { "text/plain": [ "0 0.2471774174803347\n", "1 -0.5414946268558698\n", "2 0.6278777749662021\n", "3 -0.16033855275793557\n", "4 0.7981700700350348\n", "5 -0.07503397423765741\n", "6 -0.3710273398593862\n", "dtype: object" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(str)" ] }, { "cell_type": "code", "execution_count": 19, "id": "0c4a2c37", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:44.728428Z", "iopub.status.busy": "2026-05-16T14:17:44.728147Z", "iopub.status.idle": "2026-05-16T14:17:44.731900Z", "shell.execute_reply": "2026-05-16T14:17:44.731476Z", "shell.execute_reply.started": "2026-05-16T14:17:44.728410Z" } }, "outputs": [ { "data": { "text/plain": [ "188" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(str).memory_usage()" ] }, { "cell_type": "code", "execution_count": 20, "id": "f72c5dec", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:45.170373Z", "iopub.status.busy": "2026-05-16T14:17:45.170034Z", "iopub.status.idle": "2026-05-16T14:17:45.174292Z", "shell.execute_reply": "2026-05-16T14:17:45.173860Z", "shell.execute_reply.started": "2026-05-16T14:17:45.170354Z" } }, "outputs": [ { "data": { "text/plain": [ "607" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(str).memory_usage(deep=True)" ] }, { "cell_type": "markdown", "id": "aee2d501", "metadata": {}, "source": [ "To convert to a categorical type, you can pass `\"category\"` as the type:" ] }, { "cell_type": "code", "execution_count": 21, "id": "c1dcfcfb", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:46.068389Z", "iopub.status.busy": "2026-05-16T14:17:46.068106Z", "iopub.status.idle": "2026-05-16T14:17:46.073320Z", "shell.execute_reply": "2026-05-16T14:17:46.072849Z", "shell.execute_reply.started": "2026-05-16T14:17:46.068372Z" } }, "outputs": [ { "data": { "text/plain": [ "0 0.2471774174803347\n", "1 -0.5414946268558698\n", "2 0.6278777749662021\n", "3 -0.16033855275793557\n", "4 0.7981700700350348\n", "5 -0.07503397423765741\n", "6 -0.3710273398593862\n", "dtype: category\n", "Categories (7, object): ['-0.07503397423765741', '-0.16033855275793557', '-0.3710273398593862', '-0.5414946268558698', '0.2471774174803347', '0.6278777749662021', '0.7981700700350348']" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(str).astype(\"category\")" ] }, { "cell_type": "markdown", "id": "332aecb0", "metadata": {}, "source": [ "A categorical `Series` is useful for string data and can lead to large memory savings. This is because when converting to categorical data, pandas no longer uses Python strings for each value, but repeating values are not duplicated. You still have all the features of the `str` attribute, but you save a lot of memory when you have a lot of duplicate values and you increase performance because you don’t have to do as many string operations." ] }, { "cell_type": "code", "execution_count": 22, "id": "65a2aecb", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:46.658887Z", "iopub.status.busy": "2026-05-16T14:17:46.658612Z", "iopub.status.idle": "2026-05-16T14:17:46.662728Z", "shell.execute_reply": "2026-05-16T14:17:46.662196Z", "shell.execute_reply.started": "2026-05-16T14:17:46.658869Z" } }, "outputs": [ { "data": { "text/plain": [ "495" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(\"category\").memory_usage(deep=True)" ] }, { "cell_type": "markdown", "id": "14ebb412", "metadata": {}, "source": [ "## Ordered categories\n", "\n", "To create ordered categories, you need to define your own [pandas.CategoricalDtype](https://pandas.pydata.org/docs/reference/api/pandas.CategoricalDtype.html):" ] }, { "cell_type": "code", "execution_count": 23, "id": "2aec8c09-f0f5-431c-bdb4-f7518c9ab2c1", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:47.817985Z", "iopub.status.busy": "2026-05-16T14:17:47.817409Z", "iopub.status.idle": "2026-05-16T14:17:47.824999Z", "shell.execute_reply": "2026-05-16T14:17:47.824598Z", "shell.execute_reply.started": "2026-05-16T14:17:47.817943Z" } }, "outputs": [ { "data": { "text/plain": [ "0 0.247177\n", "1 -0.541495\n", "2 0.627878\n", "3 -0.160339\n", "4 0.798170\n", "5 -0.075034\n", "6 -0.371027\n", "dtype: category\n", "Categories (7, float64): [-0.541495 < -0.371027 < -0.160339 < -0.075034 < 0.247177 < 0.627878 < 0.798170]" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pandas.api.types import CategoricalDtype\n", "\n", "\n", "s_sorted = pd.Series(sorted(set(s)))\n", "cat_dtype = CategoricalDtype(categories=s_sorted, ordered=True)\n", "\n", "s.astype(cat_dtype)" ] }, { "cell_type": "code", "execution_count": 24, "id": "ac5641a5", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:48.454293Z", "iopub.status.busy": "2026-05-16T14:17:48.453990Z", "iopub.status.idle": "2026-05-16T14:17:48.459583Z", "shell.execute_reply": "2026-05-16T14:17:48.459082Z", "shell.execute_reply.started": "2026-05-16T14:17:48.454273Z" } }, "outputs": [ { "data": { "text/plain": [ "495" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.astype(cat_dtype).memory_usage(deep=True)" ] }, { "cell_type": "markdown", "id": "cac146b8", "metadata": {}, "source": [ "The following table lists the types you can pass to `astype`.\n", "\n", "Data type | Description\n", ":-------- | :----------\n", "`str`, `'str'` | convert to Python string\n", "`'string'` | convert to Pandas string with `pandas.NA`\n", "`int`, `'int'`, `'int64'` | convert to NumPy `int64`\n", "`'int32'`, `'uint32'` | convert to NumPy `int32`\n", "`'Int64'` | convert to pandas `Int64` with `pandas.NA`\n", "`float`, `'float'`, `'float64'` | convert to floats\n", "`'category'` | convert to `CategoricalDtype` with `pandas.NA`" ] }, { "cell_type": "markdown", "id": "5bab7d2e", "metadata": {}, "source": [ "## Conversion to other data types\n", "\n", "The [pandas.Series.to_numpy](https://pandas.pydata.org/docs/reference/api/pandas.Series.to_numpy.html) method or the [pandas.Series.values](https://pandas.pydata.org/docs/reference/api/pandas.Series.values.html) property gives us a NumPy array of values, and [pandas.Series.to_list](https://pandas.pydata.org/docs/reference/api/pandas.Series.to_list.html) returns a Python list of values. Why would you want to do this? pandas objects are usually much more user-friendly and the code is easier to read. Also, python lists will be much slower to process. With [pandas.Series.to_frame](https://pandas.pydata.org/docs/reference/api/pandas.Series.to_frame.html) you can create a DataFrame with a single column, if necessary:" ] }, { "cell_type": "code", "execution_count": 25, "id": "de6e2fa8", "metadata": { "execution": { "iopub.execute_input": "2026-05-16T14:17:50.898800Z", "iopub.status.busy": "2026-05-16T14:17:50.898518Z", "iopub.status.idle": "2026-05-16T14:17:50.908589Z", "shell.execute_reply": "2026-05-16T14:17:50.907830Z", "shell.execute_reply.started": "2026-05-16T14:17:50.898782Z" } }, "outputs": [ { "data": { "text/html": [ "
| \n", " | 0 | \n", "
|---|---|
| 0 | \n", "0.247177 | \n", "
| 1 | \n", "-0.541495 | \n", "
| 2 | \n", "0.627878 | \n", "
| 3 | \n", "-0.160339 | \n", "
| 4 | \n", "0.798170 | \n", "
| 5 | \n", "-0.075034 | \n", "
| 6 | \n", "-0.371027 | \n", "