{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "\n",
    "url = \"https://de.wikipedia.org/wiki/Liste_der_Stra%C3%9Fen_und_Pl%C3%A4tze_in_Berlin-Mitte\"\n",
    "r = requests.get(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Install:\n",
    "\n",
    "   With [Spack](../../../productive/envs/spack/index.rst) you can make BeautifulSoup available in your kernel:\n",
    "\n",
    "   ``` console\n",
    "   $ spack env activate python-311\n",
    "   $ spack install py-beautifulsoup4~html5lib~lxml\n",
    "   ```\n",
    "\n",
    "   Alternatively, you can install BeautifulSoup with other package managers, for example\n",
    "\n",
    "   ``` console\n",
    "   $ uv add beautifulsoup4\n",
    "   ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. With `r.content` we can output the HTML of the page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Next, we have to decompose this string into a Python representation of the page with BeautifulSoup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "soup = BeautifulSoup(r.content, \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. To structure the code, we create a new function `get_dom` (Document Object Model) that includes all the previous code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dom(url):\n",
    "    r = request.get(url)\n",
    "    r.raise_for_status()\n",
    "    return BeautifulSoup(r.content, \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filtering out individual elements can be done, for example, via CSS selectors. These can be determined in a website, for example, by right-clicking on one of the table cells in the first column of the table in Firefox. In the Inspector that now opens, you can right-click the element again and then select *Copy → CSS Selector*. The clipboard will then contain, for example, `table.wikitable:nth-child(13) > tbody:nth-child(2) > tr:nth-child(1)`. We now clean up this CSS selector, as we do not want to filter for the 13th child element of the `table.wikitable` or the 2nd child element in `tbody`, but only for the 1st column within `tbody`.\n",
    "\n",
    "Finally, with `limit=4` in this notebook, we only display the first three results as an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<a href=\"/wiki/Ackerstra%C3%9Fe_(Berlin)\" title=\"Ackerstraße (Berlin)\">Ackerstraße</a>, <a class=\"mw-disambig\" href=\"/wiki/Adalbertstra%C3%9Fe\" title=\"Adalbertstraße\">Adalbertstraße</a>, <a class=\"new\" href=\"/w/index.php?title=Albrechtstra%C3%9Fe&amp;action=edit&amp;redlink=1\" title=\"Albrechtstraße (Seite nicht vorhanden)\">Albrechtstraße</a>, <a href=\"/wiki/Alexanderplatz\" title=\"Alexanderplatz\">Alexanderplatz</a>]\n"
     ]
    }
   ],
   "source": [
    "links = soup.select(\n",
    "    \"table.wikitable > tbody > tr > td:nth-child(1) > a\", limit=4\n",
    ")\n",
    "\n",
    "print(links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we do not want the entire HTML link, but only its text content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ackerstraße\n",
      "Adalbertstraße\n",
      "Albrechtstraße\n",
      "Alexanderplatz\n"
     ]
    }
   ],
   "source": [
    "for content in links:\n",
    "    print(content.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "**See also**\n",
    "\n",
    "* [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.13 Kernel",
   "language": "python",
   "name": "python313"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}