{ "cells": [ { "cell_type": "markdown", "id": "9f67e282", "metadata": {}, "source": [ "# XML/HTML examples" ] }, { "cell_type": "markdown", "id": "118368e4", "metadata": {}, "source": [ "## HTML\n", "\n", "Python has numerous libraries for reading and writing data in the ubiquitous HTML and XML formats. Examples are [lxml](#lxml), [Beautiful Soup](beautifulsoup.ipynb) and html5lib. While lxml is generally comparatively much faster, the other libraries are better at handling corrupted HTML or XML files.\n", "\n", "pandas has a built-in function, `read_html`, which uses libraries like lxml, html5lib and Beautiful Soup to automatically parse tables from HTML files as DataFrame objects. These have to be installed additionally. With [Spack](../../../productive/envs/spack/index.rst) you can provide lxml, BeautifulSoup and html5lib in your kernel:\n", "\n", "```console\n", "$ spack env activate python-311\n", "$ spack install py-lxml py-beautifulsoup4~html5lib~lxml py-html5lib\n", "```\n", "\n", "Alternatively, you can install BeautifulSoup with other package managers, for example\n", "\n", "```console\n", "$ uv add lxml beautifulsoup4 html5lib\n", "```" ] }, { "cell_type": "markdown", "id": "ada58b82", "metadata": {}, "source": [ "To show how this works, I use an HTML file from Wikipedia that gives an overview of different serialisation formats." ] }, { "cell_type": "code", "execution_count": 1, "id": "ee3f1a12", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "\n", "tables = pd.read_html(\"https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats\")" ] }, { "cell_type": "markdown", "id": "4e3e133c", "metadata": {}, "source": [ "The `pandas.read_html` function has a number of options, but by default it looks for and tries to parse all table data contained in `` tags. The result is a list of DataFrame objects:" ] }, { "cell_type": "code", "execution_count": 2, "id": "f12f1a55", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(tables)" ] }, { "cell_type": "code", "execution_count": 3, "id": "86ca6b3e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameCreator-maintainerBased onStandardized?[definition needed]SpecificationBinary?Human-readable?Supports references?eSchema-IDL?Standard APIsSupports zero-copy operations
0Apache AvroApache Software Foundation—NoApache Avro™ SpecificationYesPartialg—Built-inC, C#, C++, Java, PHP, Python, Ruby—
1Apache ParquetApache Software Foundation—NoApache ParquetYesNoNo—Java, Python, C++No
2Apache ThriftFacebook (creator) Apache (maintainer)—NoOriginal whitepaperYesPartialcNoBuilt-inC++, Java, Python, PHP, Ruby, Erlang, Perl, Ha...—
3ASN.1ISO, IEC, ITU-T—YesISO/IEC 8824 / ITU-T X.680 (syntax) and ISO/IE...BER, DER, PER, OER, or custom via ECNXER, JER, GSER, or custom via ECNYesfBuilt-in—OER
4BencodeBram Cohen (creator) BitTorrent, Inc. (maintai...—De facto as BEPPart of BitTorrent protocol specificationExcept numbers and delimiters, being ASCIINoNoNoNoNo
\n", "" ], "text/plain": [ " Name Creator-maintainer Based on \\\n", "0 Apache Avro Apache Software Foundation — \n", "1 Apache Parquet Apache Software Foundation — \n", "2 Apache Thrift Facebook (creator) Apache (maintainer) — \n", "3 ASN.1 ISO, IEC, ITU-T — \n", "4 Bencode Bram Cohen (creator) BitTorrent, Inc. (maintai... — \n", "\n", " Standardized?[definition needed] \\\n", "0 No \n", "1 No \n", "2 No \n", "3 Yes \n", "4 De facto as BEP \n", "\n", " Specification \\\n", "0 Apache Avro™ Specification \n", "1 Apache Parquet \n", "2 Original whitepaper \n", "3 ISO/IEC 8824 / ITU-T X.680 (syntax) and ISO/IE... \n", "4 Part of BitTorrent protocol specification \n", "\n", " Binary? \\\n", "0 Yes \n", "1 Yes \n", "2 Yes \n", "3 BER, DER, PER, OER, or custom via ECN \n", "4 Except numbers and delimiters, being ASCII \n", "\n", " Human-readable? Supports references?e Schema-IDL? \\\n", "0 Partialg — Built-in \n", "1 No No — \n", "2 Partialc No Built-in \n", "3 XER, JER, GSER, or custom via ECN Yesf Built-in \n", "4 No No No \n", "\n", " Standard APIs \\\n", "0 C, C#, C++, Java, PHP, Python, Ruby \n", "1 Java, Python, C++ \n", "2 C++, Java, Python, PHP, Ruby, Erlang, Perl, Ha... \n", "3 — \n", "4 No \n", "\n", " Supports zero-copy operations \n", "0 — \n", "1 No \n", "2 — \n", "3 OER \n", "4 No " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "formats = tables[0]\n", "\n", "formats.head()" ] }, { "cell_type": "markdown", "id": "db0585f6", "metadata": {}, "source": [ "From here we can do some [data cleansing and analysis](../../../clean-prep/index.rst), such as the number of different schema IDLs:" ] }, { "cell_type": "code", "execution_count": 4, "id": "20f41e0f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Schema-IDL?\n", "No 15\n", "Yes 5\n", "Built-in 4\n", "Schema WD 1\n", "Partial (Kwalify Archived 2021-08-12 at the Wayback Machine, Rx, built-in language type-defs) 1\n", "XML schema, RELAX NG 1\n", "WSDL, XML schema 1\n", "Partial (JSON Schema Proposal, other JSON schemas/IDLs) 1\n", "? 1\n", "Ion schema 1\n", "Partial (JSON Schema Proposal, ASN.1 with JER, Kwalify Archived 2021-08-12 at the Wayback Machine, Rx, JSON-LD 1\n", "— 1\n", "XML schema 1\n", "XML Schema 1\n", "Partial (Signature strings) 1\n", "CDDL 1\n", "Schema-IDL? 1\n", "Name: count, dtype: int64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "formats[\"Schema-IDL?\"].value_counts()" ] }, { "cell_type": "markdown", "id": "30942883", "metadata": {}, "source": [ "## XML\n", "\n", "pandas has a function `read_xml`, which makes reading XML files very easy:" ] }, { "cell_type": "code", "execution_count": 5, "id": "76f46f61", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtitlelanguageauthorlicensedate
01Python basicsenVeit SchieleBSD-3-Clause2021-10-28
12Jupyter TutorialenVeit SchieleBSD-3-Clause2019-06-27
23Jupyter TutorialdeVeit SchieleBSD-3-Clause2020-10-26
34PyViz TutorialenVeit SchieleBSD-3-Clause2020-04-13
\n", "
" ], "text/plain": [ " id title language author license date\n", "0 1 Python basics en Veit Schiele BSD-3-Clause 2021-10-28\n", "1 2 Jupyter Tutorial en Veit Schiele BSD-3-Clause 2019-06-27\n", "2 3 Jupyter Tutorial de Veit Schiele BSD-3-Clause 2020-10-26\n", "3 4 PyViz Tutorial en Veit Schiele BSD-3-Clause 2020-04-13" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_xml(\"books.xml\")" ] }, { "cell_type": "markdown", "id": "91da5bd2", "metadata": {}, "source": [ "### `lxml`\n", "\n", "Alternatively, `lxml.objectify` can be used first to parse XML files. In doing so, we get a reference to the root node of the XML file with `getroot`:" ] }, { "cell_type": "code", "execution_count": 6, "id": "391de418", "metadata": {}, "outputs": [], "source": [ "from lxml import objectify\n", "\n", "\n", "parsed = objectify.parse(open(\"books.xml\"))\n", "root = parsed.getroot()" ] }, { "cell_type": "code", "execution_count": 7, "id": "08bd89e3", "metadata": {}, "outputs": [], "source": [ "books = []\n", "\n", "for element in root.book:\n", " data = {}\n", " for child in element.getchildren():\n", " data[child.tag] = child.pyval\n", " books.append(data)" ] }, { "cell_type": "code", "execution_count": 8, "id": "074d15e5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlelanguageauthorlicensedate
0Python basicsenVeit SchieleBSD-3-Clause2021-10-28
1Jupyter TutorialenVeit SchieleBSD-3-Clause2019-06-27
2Jupyter TutorialdeVeit SchieleBSD-3-Clause2020-10-26
3PyViz TutorialenVeit SchieleBSD-3-Clause2020-04-13
\n", "
" ], "text/plain": [ " title language author license date\n", "0 Python basics en Veit Schiele BSD-3-Clause 2021-10-28\n", "1 Jupyter Tutorial en Veit Schiele BSD-3-Clause 2019-06-27\n", "2 Jupyter Tutorial de Veit Schiele BSD-3-Clause 2020-10-26\n", "3 PyViz Tutorial en Veit Schiele BSD-3-Clause 2020-04-13" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(books)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.13 Kernel", "language": "python", "name": "python313" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.0" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }