Requests installation and sample application

Installation

The requests library is useful for communicating with REST APIs. With Spack you can provide requests in your kernel:

$ spack env activate python-311
$ spack install py-requests

Alternatively, you can install requests with other package managers, for example

$ pipenv install requests

Example OSM Nominatim API

In this example we get our data from the OpenStreetMap Nominatim API. This can be reached via the URL https://nominatim.openstreetmap.org/search?. To e.g. receive information about the Berlin Congress Center in Berlin in JSON format, the URL https://nominatim.openstreetmap.org/search.php?q=Alexanderplatz+Berlin&format=json should be given, and if you want to display the corresponding map section you just have to leave out &format=json.

Then we define the base URL and the parameters. Nominatim expects at least the following two parameters

Key	Value
q	Address query that allows the following specifications: street, city, county, state, country and postalcode.
format	Format in which the data is returned. Possible values are html, xml, json, jsonv2, geojson and geocodejson.

The query can then be made with:

import requests


base_url = "https://nominatim.openstreetmap.org/search?"
params = {
    "q": "Alexanderplatz, Berlin",
    "format": "json",
}
r = requests.get(base_url, params=params)

r.status_code

200

r.json()

[{'place_id': 261767431,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'way',
  'osm_id': 783052052,
  'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],
  'lat': '52.5219814',
  'lon': '13.413635717448294',
  'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',
  'class': 'place',
  'type': 'square',
  'importance': 0.6914982526373583},
 {'place_id': 45802928,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'node',
  'osm_id': 3908141014,
  'boundingbox': ['52.5165661', '52.5265661', '13.4062804', '13.4162804'],
  'lat': '52.5215661',
  'lon': '13.4112804',
  'display_name': 'Alexanderplatz, Dircksenstraße, Mitte, Berlin, 10179, Deutschland',
  'class': 'railway',
  'type': 'station',
  'importance': 0.6560990777880803,
  'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_train_station2.p.20.png'},
 {'place_id': 190976103,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'way',
  'osm_id': 346206374,
  'boundingbox': ['52.5216214', '52.5216661', '13.4131913', '13.4131914'],
  'lat': '52.5216214',
  'lon': '13.4131913',
  'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',
  'class': 'highway',
  'type': 'pedestrian',
  'importance': 0.32000999999999996},
 {'place_id': 51760167,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'node',
  'osm_id': 4598414751,
  'boundingbox': ['52.5165849', '52.5265849', '13.4089082', '13.4189082'],
  'lat': '52.5215849',
  'lon': '13.4139082',
  'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',
  'class': 'railway',
  'type': 'station',
  'importance': 0.22000999999999998,
  'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_train_station2.p.20.png'},
 {'place_id': 49544606,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'node',
  'osm_id': 4389211800,
  'boundingbox': ['52.5231653', '52.5232653', '13.414475', '13.414575'],
  'lat': '52.5232153',
  'lon': '13.414525',
  'display_name': 'Alexanderplatz, Alexanderstraße, Mitte, Berlin, 10178, Deutschland',
  'class': 'highway',
  'type': 'bus_stop',
  'importance': 0.22000999999999998,
  'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_bus_stop2.p.20.png'},
 {'place_id': 181190866,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'way',
  'osm_id': 301241483,
  'boundingbox': ['52.5226699', '52.5227698', '13.4152008', '13.4154146'],
  'lat': '52.5227318',
  'lon': '13.4152821',
  'display_name': 'Alexanderstraße, Mitte, Berlin, 10178, Deutschland',
  'class': 'highway',
  'type': 'primary',
  'importance': 0.21000999999999995}]

Three different locations are found, the square, a bus stop and a hotel. In order to be able to filter further, we can only display the most important location:

params = {"q": "Alexanderplatz, Berlin", "format": "json", "limit": "1"}
r = requests.get(base_url, params=params)
r.json()

[{'place_id': 261767431,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'way',
  'osm_id': 783052052,
  'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],
  'lat': '52.5219814',
  'lon': '13.413635717448294',
  'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',
  'class': 'place',
  'type': 'square',
  'importance': 0.6914982526373583}]

Clean Code

Now that we know the code works, let’s turn everything into a clean and flexible function.

To ensure that the interaction was successful, we use the raise_for_status method of requests, which throws an exception if the HTTP status code isn’t 200 OK:

r.raise_for_status()

Since we don’t want to exceed the load limits of the Nominatim API, we will delay our requests with the time.sleep function:

from time import sleep


sleep(1)
r.json()

[{'place_id': 261767431,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'way',
  'osm_id': 783052052,
  'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],
  'lat': '52.5219814',
  'lon': '13.413635717448294',
  'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',
  'class': 'place',
  'type': 'square',
  'importance': 0.6914982526373583}]

Next we declare the function itself. As arguments we need the address, the format, the limit of the objects to be returned with the default value 1 and further kwargs (keyword arguments) that are passed as parameters:

def nominatim_search(address, format="json", limit=1, **kwargs):
    """Thin wrapper around the Nominatim search API.
    For the list of parameters see
    https://nominatim.org/release-docs/develop/api/Search/#parameters
    """
    search_url = "https://nominatim.openstreetmap.org/search?"
    params = {"q": address, "format": format, "limit": limit, **kwargs}
    r = requests.get(search_url, params=params)
    # Raise an exception if the status is unsuccessful
    r.raise_for_status()

    sleep(1)
    return r.json()

Now we can try out the function, for example with

nominatim_search("Alexanderplatz, Berlin")

[{'place_id': 261767431,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'way',
  'osm_id': 783052052,
  'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],
  'lat': '52.5219814',
  'lon': '13.413635717448294',
  'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',
  'class': 'place',
  'type': 'square',
  'importance': 0.6914982526373583}]

However, you can use other parameters besides address. You can get an overview in the Nominatim Docs.

nominatim_search(
    address=None,
    street="8, Marienburger Straße",
    city="Berlin",
    country="Germany",
)

[{'place_id': 194632110,
  'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',
  'osm_type': 'way',
  'osm_id': 368946992,
  'boundingbox': ['52.5341829', '52.534317', '13.4237697', '13.4240606'],
  'lat': '52.53425085',
  'lon': '13.423921421750569',
  'display_name': '8, Marienburger Straße, Winsviertel, Prenzlauer Berg, Pankow, Berlin, 10405, Deutschland',
  'class': 'building',
  'type': 'apartments',
  'importance': 0.4200099999999999}]

Caching

If the same queries are to be asked over and over again within a session, it makes sense to call up this data only once and use it again. In Python we can use lru_cache from Python’s standard functools library. lru_cache saves the last N requests (Least Recent Used) and as soon as the limit is exceeded, the oldest values are discarded. To use this for the nominatim_search method, all you have to do is define an import and a decorator:

from functools import lru_cache


@lru_cache(maxsize=1000)
def nominatim_search(address, format="json", limit=1, **kwargs):
    """…"""

However, lru_cache only saves the results during a session. If a script terminates because of a timeout or an exception, the results are lost. If the data is to be saved more permanently, tools such as joblib or python-diskcache can be used.