{ "cells": [ { "cell_type": "markdown", "id": "fbfdbb51", "metadata": {}, "source": [ "# Manipulation of strings\n", "\n", "pandas offers the possibility to concisely apply Python’s string methods and regular expressions to whole arrays of data.\n", "\n", "
\n", "\n", "**See also:**\n", "\n", "* [Strings](https://python-basics-tutorial.readthedocs.io/en/latest/types/strings.html)\n", "* [re](https://python-basics-tutorial.readthedocs.io/en/latest/types/strings.html#re)\n", "
" ] }, { "cell_type": "markdown", "id": "6e83e2cc", "metadata": {}, "source": [ "## Vectorised string functions in pandas\n", "\n", "Cleaning up a cluttered dataset for analysis often requires a lot of string manipulation. To make matters worse, a column containing strings sometimes has missing data:" ] }, { "cell_type": "code", "execution_count": 1, "id": "9bd032ba", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Veit NaN\n", "Veit Schiele veit.schiele@cusy.io\n", "cusy GmbH info@cusy.io\n", "dtype: object" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "\n", "addresses = {\n", " \"Veit\": np.nan,\n", " \"Veit Schiele\": \"veit.schiele@cusy.io\",\n", " \"cusy GmbH\": \"info@cusy.io\",\n", "}\n", "addresses = pd.Series(addresses)\n", "\n", "addresses" ] }, { "cell_type": "code", "execution_count": 2, "id": "9396bd11", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Veit True\n", "Veit Schiele False\n", "cusy GmbH False\n", "dtype: bool" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "addresses.isna()" ] }, { "cell_type": "markdown", "id": "86160b32", "metadata": {}, "source": [ "You can apply string and regular expression methods to any value (by passing a lambda or other function) using `data.map`, but this fails for `NA` values. To deal with this, `Series` has array-oriented methods for string operations that skip and pass `NA` values. These are accessed via Series’ `str` attribute; for example, we could use `str.contains` to check whether each email address contains `veit`:" ] }, { "cell_type": "code", "execution_count": 3, "id": "21cf088b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Veit NaN\n", "Veit Schiele True\n", "cusy GmbH False\n", "dtype: object" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "addresses.str.contains(\"veit\")" ] }, { "cell_type": "markdown", "id": "bdf32408", "metadata": {}, "source": [ "Regular expressions can also be used, along with options such as `IGNORECASE`:" ] }, { "cell_type": "code", "execution_count": 4, "id": "29f1fd91", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Veit NaN\n", "Veit Schiele (veit.schiele, cusy, io)\n", "cusy GmbH (info, cusy, io)\n", "dtype: object" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import re\n", "\n", "\n", "pattern = r\"([A-Z0-9._%+-]+)@([A-Z0-9.-]+)\\.([A-Z]{2,4})\"\n", "matches = addresses.str.findall(pattern, flags=re.IGNORECASE).str[0]\n", "\n", "matches" ] }, { "cell_type": "markdown", "id": "5aa3c315", "metadata": {}, "source": [ "There are several ways to retrieve a vectorised element. Either use `str.get` or the index of `str`:" ] }, { "cell_type": "code", "execution_count": 5, "id": "5707817e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Veit NaN\n", "Veit Schiele cusy\n", "cusy GmbH cusy\n", "dtype: object" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "matches.str.get(1)" ] }, { "cell_type": "markdown", "id": "b541b624", "metadata": {}, "source": [ "Similarly, you can also cut strings with this syntax:" ] }, { "cell_type": "code", "execution_count": 6, "id": "dc29f6a0", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Veit NaN\n", "Veit Schiele veit.\n", "cusy GmbH info@\n", "dtype: object" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "addresses.str[:5]" ] }, { "cell_type": "markdown", "id": "c44a297f", "metadata": {}, "source": [ "The [pandas.Series.str.extract](https://pandas.pydata.org/docs/reference/api/pandas.Series.str.extract.html) method returns the captured groups of a regular expression as a DataFrame:" ] }, { "cell_type": "code", "execution_count": 7, "id": "83e94706", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
VeitNaNNaNNaN
Veit Schieleveit.schielecusyio
cusy GmbHinfocusyio
\n", "
" ], "text/plain": [ " 0 1 2\n", "Veit NaN NaN NaN\n", "Veit Schiele veit.schiele cusy io\n", "cusy GmbH info cusy io" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "addresses.str.extract(pattern, flags=re.IGNORECASE)" ] }, { "cell_type": "markdown", "id": "3b44535b", "metadata": {}, "source": [ "More vectorised pandas string methods:\n", "\n", "Method | Description\n", ":----- | :----------\n", "`cat` | concatenates strings element by element with optional delimiter\n", "`contains` | returns a boolean array if each string contains a pattern/gex\n", "`count` | counts occurrences of the pattern\n", "`extract` | uses a regular expression with groups to extract one or more strings from a set of strings; the result is a DataFrame with one column per group\n", "`endswith` | equivalent to `x.endswith(pattern)` for each element\n", "`startswith` | equivalent to `x.startswith(pattern)` for each element\n", "`findall` | computes list of all occurrences of pattern/regex for each string\n", "`get` | index in each element (get `i`-th element)\n", "`isalnum` | Equivalent to built-in `str.alnum`\n", "`isalpha` | Equivalent to built-in `str.isalpha`\n", "`isdecimal` | Equivalent to built-in `str.isdecimal`\n", "`isdigit` | Equivalent to built-in `str.isdigit`\n", "`islower` | Equivalent to built-in `str.islower`\n", "`isnumeric` | Equivalent to built-in `str.isnumeric`\n", "`isupper` | Equivalent to built-in `str.isupper`\n", "`join` | joins strings in each element of the series with the passed separator character\n", "`len` | calculates the length of each string\n", "`lower`, `upper` | converts case; equivalent to `x.lower()` or `x.upper()` for each element\n", "`match` | uses `re.match` with the passed regular expression for each element, returning `True` or `False` if matched.\n", "`extract` | captures group elements (if any) by index from each string\n", "`pad` | inserts spaces on the left, right or both sides of strings\n", "`centre` | Equivalent to `pad(side='both')`\n", "`repeat` | Duplicates values (for example `s.str.repeat(3)` equals `x * 3` for each string)\n", "`replace` | replaces pattern/rulex with another string\n", "`slice` | splits each string in the series\n", "`split` | splits strings using delimiters or regular expressions\n", "`strip` | truncates spaces on both sides, including line breaks\n", "`rstrip` | truncates spaces on the right side\n", "`lstrip` | truncates spaces on the left side" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.13 Kernel", "language": "python", "name": "python313" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.0" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }