{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# BeautifulSoup"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-19T21:26:50.284314Z",
"iopub.status.busy": "2026-05-19T21:26:50.283914Z",
"iopub.status.idle": "2026-05-19T21:26:50.468784Z",
"shell.execute_reply": "2026-05-19T21:26:50.468448Z",
"shell.execute_reply.started": "2026-05-19T21:26:50.284288Z"
}
},
"outputs": [],
"source": [
"import httpx\n",
"\n",
"\n",
"url = \"https://docs.python.org/3/library/xml.dom.html\"\n",
"r = httpx.get(url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Install:\n",
"\n",
" With [Spack](../../../productive/envs/spack/index.rst) you can make BeautifulSoup available in your kernel:\n",
"\n",
" ``` console\n",
" $ spack env activate python-313\n",
" $ spack install py-beautifulsoup4~html5lib~lxml\n",
" ```\n",
"\n",
" Alternatively, you can install BeautifulSoup with other package managers, for example\n",
"\n",
" ``` console\n",
" $ uv add beautifulsoup4\n",
" ```\n",
"\n",
"2. With `r.content` we can output the HTML of the page.\n",
"\n",
"3. Next, we have to decompose this string into a Python representation of the page with BeautifulSoup:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-19T21:26:50.469199Z",
"iopub.status.busy": "2026-05-19T21:26:50.469108Z",
"iopub.status.idle": "2026-05-19T21:26:50.536394Z",
"shell.execute_reply": "2026-05-19T21:26:50.536080Z",
"shell.execute_reply.started": "2026-05-19T21:26:50.469191Z"
},
"scrolled": true
},
"outputs": [],
"source": [
"from bs4 import BeautifulSoup\n",
"\n",
"\n",
"soup = BeautifulSoup(r.content, \"html.parser\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4. To structure the code, we create a new function `get_dom` (Document Object Model) that includes all the previous code:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-19T21:26:50.537018Z",
"iopub.status.busy": "2026-05-19T21:26:50.536890Z",
"iopub.status.idle": "2026-05-19T21:26:50.539152Z",
"shell.execute_reply": "2026-05-19T21:26:50.538815Z",
"shell.execute_reply.started": "2026-05-19T21:26:50.537009Z"
}
},
"outputs": [],
"source": [
"def get_dom(url):\n",
" r = httpx.get(url)\n",
" r.raise_for_status()\n",
" return BeautifulSoup(r.content, \"html.parser\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Filtering out individual elements can be done, for example, via CSS selectors. These can be determined in a website, for example, by right-clicking on one of the table cells in the first column of the table in Firefox. In the Inspector that now opens, you can right-click the element again and then select *Copy → CSS Selector*. The clipboard will then contain, for example, `table.wikitable:nth-child(13) > tbody:nth-child(2) > tr:nth-child(1)`. We now clean up this CSS selector, as we do not want to filter for the 13th child element of the `table.wikitable` or the 2nd child element in `tbody`, but only for the 1st column within `tbody`.\n",
"\n",
"Finally, with `limit=4` in this notebook, we only display the first three results as an example:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-19T21:26:50.539666Z",
"iopub.status.busy": "2026-05-19T21:26:50.539527Z",
"iopub.status.idle": "2026-05-19T21:26:50.543325Z",
"shell.execute_reply": "2026-05-19T21:26:50.543085Z",
"shell.execute_reply.started": "2026-05-19T21:26:50.539657Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[DOMImplementation, Node, NodeList, DocumentType]\n"
]
}
],
"source": [
"links = soup.select(\n",
" \"#objects-in-the-dom > table > tbody > tr > td > p > code .pre\",\n",
" limit=4,\n",
")\n",
"\n",
"print(links)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, we do not want the entire HTML link, but only its text content:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-19T21:26:50.543804Z",
"iopub.status.busy": "2026-05-19T21:26:50.543726Z",
"iopub.status.idle": "2026-05-19T21:26:50.545409Z",
"shell.execute_reply": "2026-05-19T21:26:50.545122Z",
"shell.execute_reply.started": "2026-05-19T21:26:50.543797Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"DOMImplementation\n",
"Node\n",
"NodeList\n",
"DocumentType\n"
]
}
],
"source": [
"for content in links:\n",
" print(content.text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"**See also**\n",
"\n",
"* [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n",
"
"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.13 Kernel",
"language": "python",
"name": "python313"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.0"
},
"latex_envs": {
"LaTeX_envs_menu_present": true,
"autoclose": false,
"autocomplete": true,
"bibliofile": "biblio.bib",
"cite_by": "apalike",
"current_citInitial": 1,
"eqLabelWithNumbers": true,
"eqNumInitial": 1,
"hotkeys": {
"equation": "Ctrl-E",
"itemize": "Ctrl-I"
},
"labels_anchors": false,
"latex_user_defs": false,
"report_style_numbering": false,
"user_envs_cfg": false
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}