Text analysis

Encoding

See also:

Problematic control characters

At first glance, it may seem sensible to allow any Unicode characters in character strings. However, this is rarely advisable. RFC 9839: Unicode Character Repertoire Subsets defines three classes of problematic code points: legacy controls, non-characters and surrogate characters. They can lead to unclear or confusing data analyses or cause processing problems and should therefore be flagged, replaced or removed.

Specify text encoding

When reading in text, specify the correct text encoding:

[1]:

text = "El Niño"

text.encode("utf-8")

[1]:

b'El Ni\xc3\xb1o'

Normalising the text

The Python library charset-normalizer can help you determine the correct encodings. You can use the library, for example, as follows:

uv add charset-normalizer

[2]:

!normalizer ../../../data/iot_example.json

{
    "path": "/Users/veit/cusy/trn/Python4DataScience/data/iot_example.json",
    "encoding": "ascii",
    "encoding_aliases": [
        "646",
        "ansi_x3.4_1968",
        "ansi_x3_4_1968",
        "ansi_x3.4_1986",
        "cp367",
        "csascii",
        "ibm367",
        "iso646_us",
        "iso_646.irv_1991",
        "iso_ir_6",
        "us",
        "us_ascii"
    ],
    "alternative_encodings": [],
    "language": "English",
    "alphabets": [
        "Basic Latin",
        "Control character"
    ],
    "has_sig_or_bom": false,
    "chaos": 0.0,
    "coherence": 0.0,
    "unicode_path": null,
    "is_preferred": true
}

or

[3]:

from charset_normalizer import from_path


iot_example = from_path("../../../data/iot_example.json")
print(str(iot_example.best()))

{
    "creation_metadata": {
        "local_time": "2026-09-18T13:15:32",
        "utc_time": "2026-09-18T11:15:32+00:00",
        "creator": "TDDA 3.0.01",
        "host": "fay.local",
        "user": "veit",
        "n_records": 146397,
        "n_selected": 146397
    },
    "fields": {
        "timestamp": {
            "type": "string",
            "min_length": 19,
            "max_length": 19,
            "max_nulls": 0,
            "no_duplicates": true
        },
        "username": {
            "type": "string",
            "min_length": 3,
            "max_length": 21,
            "max_nulls": 0
        },
        "temperature": {
            "type": "int",
            "min": 5,
            "max": 29,
            "sign": "positive",
            "max_nulls": 0
        },
        "heartrate": {
            "type": "int",
            "min": 60,
            "max": 89,
            "sign": "positive",
            "max_nulls": 0
        },
        "build": {
            "type": "string",
            "min_length": 36,
            "max_length": 36,
            "max_nulls": 0,
            "no_duplicates": true
        },
        "latest": {
            "type": "int",
            "min": 0,
            "max": 1,
            "sign": "non-negative",
            "max_nulls": 0
        },
        "note": {
            "type": "string",
            "min_length": 4,
            "max_length": 8,
            "allowed_values": [
                "interval",
                "sleep",
                "test",
                "update",
                "user",
                "wake"
            ]
        }
    },
    "dataset": {
        "allowed_fields": [],
        "required_fields": [
            "*"
        ]
    }
}

However, not all text is encoded unambiguously:

Assume all external input is the result of (a series of) bugs.

– RFC 9225

This is where ftfy can help you:

uv add ftfy

[4]:

import ftfy


ftfy.fix_text("l’humanité")

[4]:

"l'humanité"

Usually, normalisation is carried out to the Normalisation Form Canonical Composition (NFC) or the Normalisation Form Compatibility Composition (NFKC):

[5]:

import unicodedata


nfkc = unicodedata.normalize("NFKC", text)

"; ".join(f"U+{ord(c):04X}: {unicodedata.name(c)}" for c in nfkc)

[5]:

'U+0045: LATIN CAPITAL LETTER E; U+006C: LATIN SMALL LETTER L; U+0020: SPACE; U+004E: LATIN CAPITAL LETTER N; U+0069: LATIN SMALL LETTER I; U+00F1: LATIN SMALL LETTER N WITH TILDE; U+006F: LATIN SMALL LETTER O'

Before you compare texts, both should be in the same normalised form:

[6]:

nfkd = unicodedata.normalize("NFKD", text)

nfkc == nfkd

[6]:

False

The length of the texts also depends on the normalisation form:

[7]:

len(nfkc) == len(nfkd)

[7]:

False

See also:

String-Matching

difflib

String-searching algorithm are used to find text segments within a string based on a search pattern. In Python, for example, search patterns can be specified using the re module.

However, you can also compare strings using the get_close_matches option in Python’s difflib library.

[8]:

import difflib


options = [
    "Berlin",
    "Berlin, Germany",
    "Berlin, Deutschland",
    "Berlin, DE",
    "Bundeshauptstadt",
    "Spreeathen",
]

difflib.get_close_matches("Brln", options)

[8]:

['Berlin']

If you want to specify the minimum match score required for a suggestion to be displayed, you can change the default value of the cutoff argument from 0.6:

[9]:

difflib.get_close_matches("Brln", options, cutoff=0.5)

[9]:

['Berlin', 'Berlin, DE']

Note:

In Python 3.14, argparse gained a new option, suggest_on_error, which is based precisely on this.

Using the difflib SequenceMatcher, we can also calculate the similarity between two strings:

[10]:

from difflib import SequenceMatcher


m = SequenceMatcher(None, "Brln", options[0])

m.ratio()

[10]:

0.8

The two strings Brlin and Berlin appear to be 80 per cent identical. Whilst the standard measure of string similarity works well for individual words and long strings, it is less suitable for short strings containing between two and ten words. The naive approach is far too sensitive to minor differences in word order, missing or extra words, and other such issues:

[11]:

m = SequenceMatcher(None, "Brln", options[1])

m.ratio()

[11]:

0.42105263157894735

TheFuzz

TheFuzz goes beyond the capabilities of difflib. The various methods available and the differences between them are described in the blog post FuzzyWuzzy: Fuzzy String Matching in Python.

1. Installation

You can use uv to make TheFuzz and the optional python-levenshtein library available in your kernel:

$ uv add thefuzz

2. Import

[12]:

from thefuzz import fuzz, process

3. Example

3.1 String similarity

Here, too, calculating string similarity initially yields the same result:

[13]:

fuzz.ratio("Brln", options[1])

[13]:

42
3.2 Partial string similarity

However, we can also use a heuristic here known as best partial.

[14]:

fuzz.partial_ratio("Brln", options[1])

[14]:

75
3.3 Token Sorting

In token sorting, the string in question is broken down into tokens, the tokens are sorted alphabetically and then reassembled into a string, for example:

[15]:

fuzz.token_set_ratio("Brln", options[1])

[15]:

44
3.4 Extracting from a list
[16]:

process.extract("Brln", options, limit=1)

[16]:

[('Berlin', 80)]

[17]:

process.extract("Brln", options)

[17]:

[('Berlin', 80),
 ('Berlin, Germany', 68),
 ('Berlin, Deutschland', 68),
 ('Berlin, DE', 68),
 ('Bundeshauptstadt', 51)]

4. Further information

[18]:

process.extract?

Signature:
process.extract(
    query,
    choices,
    processor=<function full_process at 0x114722c00>,
    scorer=<function WRatio at 0x1147234c0>,
    limit=5,
)
Docstring:
Select the best match in a list or dictionary of choices.

Find best matches in a list or dictionary of choices, return a
list of tuples containing the match and its score. If a dictionary
is used, also returns the key for each match.

Arguments:
    query: An object representing the thing we want to find.
    choices: An iterable or dictionary-like object containing choices
        to be matched against the query. Dictionary arguments of
        {key: value} pairs will attempt to match the query against
        each value.
    processor: Optional function of the form f(a) -> b, where a is the query or
        individual choice and b is the choice to be used in matching.

        This can be used to match against, say, the first element of
        a list:

        lambda x: x[0]

        Defaults to thefuzz.utils.full_process().
    scorer: Optional function for scoring matches between the query and
        an individual processed choice. This should be a function
        of the form f(query, choice) -> int.
        By default, fuzz.WRatio() is used and expects both query and
        choice to be strings.
    limit: Optional maximum for the number of elements returned. Defaults
        to 5.

Returns:
    List of tuples containing the match and its score.

    If a list is used for choices, then the result will be 2-tuples.
    If a dictionary is used, then the result will be 3-tuples containing
    the key for each match.

    For example, searching for 'bird' in the dictionary

    {'bard': 'train', 'dog': 'man'}

    may return

    [('train', 22, 'bard'), ('man', 0, 'dog')]
File:      ~/cusy/trn/jupyter-tutorial/uvenvs/py313/.venv/lib/python3.13/site-packages/thefuzz/process.py
Type:      function

See also:

Typical algorithms for string matching are:

There are also algorithms for words that sound similar:

5. Known ports

FuzzyWuzzy is also being ported to other languages! Here are some known ports:

textacy

textacy uses the spaCy library for tasks such as tokenisation, part-of-speech tagging and dependency parsing.

You can access and extend spaCy’s core functionality via convenient methods and custom extensions, allowing you to easily analyse one or more documents:

  1. Load prepared datasets containing text content and metadata

  2. Cleaning, normalising and examining text content in its raw form before further processing with spaCy

  3. Extracting structured information from processed documents, including n-grams, entities, acronyms, identification keys and SVO triples.

  4. Comparing strings and sequences using various similarity metrics

  5. Tokenising and vectorising documents, which can then be used to train, interpret and visualise topic models

  6. Calculating statistics on the readability and lexical diversity of texts, including the Flesch-Kincaid Readability Index, the multilingual Flesch Reading Ease Index and the type-token ratio