String comparisons¶

`difflib`¶

You can compare strings using the get_close_matches option of Python’s built-in difflib library.

[1]:

import difflib


options = [
    "Berlin",
    "Berlin, Germany",
    "Berlin, Deutschland",
    "Berlin, DE",
    "Bundeshauptstadt",
    "Spreeathen",
]

difflib.get_close_matches("Brln", options)

[1]:

['Berlin']

If you want to specify the minimum match score required for a suggestion to be displayed, you can change the default value of the cutoff argument from 0.6:

[2]:

difflib.get_close_matches("Brln", options, cutoff=0.5)

[2]:

['Berlin', 'Berlin, DE']

Note:

In Python 3.14, argparse gained a new option, suggest_on_error, which is based precisely on this.

Using the difflib SequenceMatcher, we can also calculate the similarity between two strings:

[3]:

from difflib import SequenceMatcher

m = SequenceMatcher(None, "Brln", options[0])

m.ratio()

[3]:

0.8

The two strings Brlin and Berlin appear to be 80% identical. Whilst the standard measure of string similarity works well for individual words and long strings, it is less suitable for short strings containing between two and ten words. The naive approach is far too sensitive to minor differences in word order, missing or extra words, and other such issues:

[4]:

m = SequenceMatcher(None, "Brln", options[1])

m.ratio()

[4]:

0.42105263157894735

`TheFuzz`¶

TheFuzz goes beyond the capabilities of difflib. The various methods available and the differences between them are described in the blog post FuzzyWuzzy: Fuzzy String Matching in Python.

1. Installation¶

You can use uv to make TheFuzz and the optional python-levenshtein library available in your kernel:

$ uv add thefuzz

2. Import¶

[5]:

from thefuzz import fuzz, process

3. Example¶

3.1 String similarity¶

Here, too, the calculation of string similarity initially yields the same result:

[6]:

fuzz.ratio("Brln", options[1])

[6]:

3.2 Partial string similarity¶

However, we can also use a heuristic here known as best partial.

[7]:

fuzz.partial_ratio("Brln", options[1])

[7]:

3.3 Token sorting¶

In token sorting, the string in question is broken down into tokens, the tokens are sorted alphabetically and then reassembled into a string, for example:

[8]:

fuzz.token_set_ratio("Brln", options[1])

[8]:

3.4 Extracting from a list¶

[9]:

process.extract("Brln", options, limit=1)

[9]:

[('Berlin', 80)]

[10]:

process.extract("Brln", options)

[10]:

[('Berlin', 80),
 ('Berlin, Germany', 68),
 ('Berlin, Deutschland', 68),
 ('Berlin, DE', 68),
 ('Bundeshauptstadt', 51)]

4. Further information¶

[11]:

process.extract?

Signature:
process.extract(
    query,
    choices,
    processor=<function full_process at 0x105c2f380>,
    scorer=<function WRatio at 0x130c663e0>,
    limit=5,
)
Docstring:
Select the best match in a list or dictionary of choices.

Find best matches in a list or dictionary of choices, return a
list of tuples containing the match and its score. If a dictionary
is used, also returns the key for each match.

Arguments:
    query: An object representing the thing we want to find.
    choices: An iterable or dictionary-like object containing choices
        to be matched against the query. Dictionary arguments of
        {key: value} pairs will attempt to match the query against
        each value.
    processor: Optional function of the form f(a) -> b, where a is the query or
        individual choice and b is the choice to be used in matching.

        This can be used to match against, say, the first element of
        a list:

        lambda x: x[0]

        Defaults to thefuzz.utils.full_process().
    scorer: Optional function for scoring matches between the query and
        an individual processed choice. This should be a function
        of the form f(query, choice) -> int.
        By default, fuzz.WRatio() is used and expects both query and
        choice to be strings.
    limit: Optional maximum for the number of elements returned. Defaults
        to 5.

Returns:
    List of tuples containing the match and its score.

    If a list is used for choices, then the result will be 2-tuples.
    If a dictionary is used, then the result will be 3-tuples containing
    the key for each match.

    For example, searching for 'bird' in the dictionary

    {'bard': 'train', 'dog': 'man'}

    may return

    [('train', 22, 'bard'), ('man', 0, 'dog')]
File:      ~/.local/share/virtualenvs/python-311-6zxVKbDJ/lib/python3.11/site-packages/thefuzz/process.py
Type:      function

5. Known ports¶

FuzzyWuzzy is also being ported to other languages! Here are some known ports:

textacy¶

textacy uses the spaCy library for tasks such as tokenisation, part-of-speech tagging and dependency parsing.

You can access and extend spaCy’s core functionality via convenient methods and custom extensions, allowing you to easily analyse one or more documents:

Load prepared datasets containing text content and metadata
Cleaning, normalising and examining text content in its raw form before further processing with spaCy
Extracting structured information from processed documents, including n-grams, entities, acronyms, identification keys and SVO triples.
Comparing strings and sequences using various similarity metrics
Tokenising and vectorising documents, which can then be used to train, interpret and visualise topic models
Calculating statistics on the readability and lexical diversity of texts, including the Flesch-Kincaid Readability Index, the multilingual Flesch Reading Ease Index and the type-token ratio