{ "cells": [ { "cell_type": "markdown", "id": "9f67e282", "metadata": {}, "source": [ "# XML/HTML examples" ] }, { "cell_type": "markdown", "id": "118368e4", "metadata": {}, "source": [ "## HTML\n", "\n", "Python has numerous libraries for reading and writing data in the ubiquitous HTML and XML formats. Examples are [lxml](#lxml), [Beautiful Soup](beautifulsoup.ipynb) and html5lib. While lxml is generally comparatively much faster, the other libraries are better at handling corrupted HTML or XML files.\n", "\n", "pandas has a built-in function, `read_html`, which uses libraries like lxml, html5lib and Beautiful Soup to automatically parse tables from HTML files as DataFrame objects. These have to be installed additionally. With [Spack](../../../productive/envs/spack/index.rst) you can provide lxml, BeautifulSoup and html5lib in your kernel:\n", "\n", "```console\n", "$ spack env activate python-311\n", "$ spack install py-lxml py-beautifulsoup4~html5lib~lxml py-html5lib\n", "```\n", "\n", "Alternatively, you can install BeautifulSoup with other package managers, for example\n", "\n", "```console\n", "$ uv add lxml beautifulsoup4 html5lib\n", "```" ] }, { "cell_type": "markdown", "id": "ada58b82", "metadata": {}, "source": [ "To show how this works, I use an HTML file from Wikipedia that gives an overview of different serialisation formats." ] }, { "cell_type": "code", "execution_count": 1, "id": "ee3f1a12", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.186839Z", "iopub.status.busy": "2026-05-19T20:56:26.186656Z", "iopub.status.idle": "2026-05-19T20:56:26.576094Z", "shell.execute_reply": "2026-05-19T20:56:26.575655Z", "shell.execute_reply.started": "2026-05-19T20:56:26.186821Z" } }, "outputs": [], "source": [ "import pandas as pd\n", "\n", "\n", "tables = pd.read_html(\n", " \"https://docs.python.org/3/library/xml.dom.html\",\n", ")" ] }, { "cell_type": "markdown", "id": "4e3e133c", "metadata": {}, "source": [ "The `pandas.read_html` function has a number of options, but by default it looks for and tries to parse all table data contained in `` tags. The result is a list of DataFrame objects:" ] }, { "cell_type": "code", "execution_count": 2, "id": "f12f1a55", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.576717Z", "iopub.status.busy": "2026-05-19T20:56:26.576552Z", "iopub.status.idle": "2026-05-19T20:56:26.582626Z", "shell.execute_reply": "2026-05-19T20:56:26.582265Z", "shell.execute_reply.started": "2026-05-19T20:56:26.576706Z" } }, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(tables)" ] }, { "cell_type": "code", "execution_count": 3, "id": "86ca6b3e", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.583274Z", "iopub.status.busy": "2026-05-19T20:56:26.583142Z", "iopub.status.idle": "2026-05-19T20:56:26.587950Z", "shell.execute_reply": "2026-05-19T20:56:26.587664Z", "shell.execute_reply.started": "2026-05-19T20:56:26.583259Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDL TypePython Type
0booleanbool or int
1intint
2long intint
3unsigned intint
4DOMStringstr or bytes
\n", "" ], "text/plain": [ " IDL Type Python Type\n", "0 boolean bool or int\n", "1 int int\n", "2 long int int\n", "3 unsigned int int\n", "4 DOMString str or bytes" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xml_idl = tables[2]\n", "\n", "xml_idl.head()" ] }, { "cell_type": "markdown", "id": "db0585f6", "metadata": {}, "source": [ "From here we can do some [data cleansing and analysis](../../../clean-prep/index.rst), such as the number of different Python types:" ] }, { "cell_type": "code", "execution_count": 4, "id": "20f41e0f", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.588418Z", "iopub.status.busy": "2026-05-19T20:56:26.588339Z", "iopub.status.idle": "2026-05-19T20:56:26.591023Z", "shell.execute_reply": "2026-05-19T20:56:26.590781Z", "shell.execute_reply.started": "2026-05-19T20:56:26.588411Z" } }, "outputs": [ { "data": { "text/plain": [ "Python Type\n", "int 3\n", "bool or int 1\n", "str or bytes 1\n", "Name: count, dtype: int64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xml_idl[\"Python Type\"].value_counts()" ] }, { "cell_type": "markdown", "id": "30942883", "metadata": {}, "source": [ "## XML\n", "\n", "pandas has a function `read_xml`, which makes reading XML files very easy:" ] }, { "cell_type": "code", "execution_count": 5, "id": "76f46f61", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.591373Z", "iopub.status.busy": "2026-05-19T20:56:26.591300Z", "iopub.status.idle": "2026-05-19T20:56:26.595950Z", "shell.execute_reply": "2026-05-19T20:56:26.595725Z", "shell.execute_reply.started": "2026-05-19T20:56:26.591366Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtitlelanguageauthorlicensedate
01Python basicsenVeit SchieleBSD-3-Clause2021-10-28
12Jupyter TutorialenVeit SchieleBSD-3-Clause2019-06-27
23Jupyter TutorialdeVeit SchieleBSD-3-Clause2020-10-26
34PyViz TutorialenVeit SchieleBSD-3-Clause2020-04-13
\n", "
" ], "text/plain": [ " id title language author license date\n", "0 1 Python basics en Veit Schiele BSD-3-Clause 2021-10-28\n", "1 2 Jupyter Tutorial en Veit Schiele BSD-3-Clause 2019-06-27\n", "2 3 Jupyter Tutorial de Veit Schiele BSD-3-Clause 2020-10-26\n", "3 4 PyViz Tutorial en Veit Schiele BSD-3-Clause 2020-04-13" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_xml(\"books.xml\")" ] }, { "cell_type": "markdown", "id": "91da5bd2", "metadata": {}, "source": [ "### `lxml`\n", "\n", "Alternatively, `lxml.objectify` can be used first to parse XML files. In doing so, we get a reference to the root node of the XML file with `getroot`:" ] }, { "cell_type": "code", "execution_count": 6, "id": "391de418", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.597376Z", "iopub.status.busy": "2026-05-19T20:56:26.597281Z", "iopub.status.idle": "2026-05-19T20:56:26.604652Z", "shell.execute_reply": "2026-05-19T20:56:26.604407Z", "shell.execute_reply.started": "2026-05-19T20:56:26.597369Z" } }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "from lxml import objectify\n", "\n", "\n", "parsed = objectify.parse(Path.open(\"books.xml\"))\n", "root = parsed.getroot()" ] }, { "cell_type": "code", "execution_count": 7, "id": "08bd89e3", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.605121Z", "iopub.status.busy": "2026-05-19T20:56:26.605036Z", "iopub.status.idle": "2026-05-19T20:56:26.607410Z", "shell.execute_reply": "2026-05-19T20:56:26.607198Z", "shell.execute_reply.started": "2026-05-19T20:56:26.605107Z" } }, "outputs": [], "source": [ "books = []\n", "\n", "for element in root.book:\n", " data = {}\n", " for child in element.getchildren():\n", " data[child.tag] = child.pyval\n", " books.append(data)" ] }, { "cell_type": "code", "execution_count": 8, "id": "074d15e5", "metadata": { "execution": { "iopub.execute_input": "2026-05-19T20:56:26.607921Z", "iopub.status.busy": "2026-05-19T20:56:26.607779Z", "iopub.status.idle": "2026-05-19T20:56:26.611104Z", "shell.execute_reply": "2026-05-19T20:56:26.610891Z", "shell.execute_reply.started": "2026-05-19T20:56:26.607913Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlelanguageauthorlicensedate
0Python basicsenVeit SchieleBSD-3-Clause2021-10-28
1Jupyter TutorialenVeit SchieleBSD-3-Clause2019-06-27
2Jupyter TutorialdeVeit SchieleBSD-3-Clause2020-10-26
3PyViz TutorialenVeit SchieleBSD-3-Clause2020-04-13
\n", "
" ], "text/plain": [ " title language author license date\n", "0 Python basics en Veit Schiele BSD-3-Clause 2021-10-28\n", "1 Jupyter Tutorial en Veit Schiele BSD-3-Clause 2019-06-27\n", "2 Jupyter Tutorial de Veit Schiele BSD-3-Clause 2020-10-26\n", "3 PyViz Tutorial en Veit Schiele BSD-3-Clause 2020-04-13" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(books)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.13 Kernel", "language": "python", "name": "python313" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.0" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }