{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# TDDA: Test-Driven Data Analysis\n", "\n", "[TDDA](https://github.com/tdda/tdda) uses file inputs (such as NumPy arrays or Pandas DataFrames) and a set of constraints that are stored as a JSON file.\n", "\n", "* [Reference Tests](https://tdda.readthedocs.io/en/latest/referencetest.html) supports the creation of reference tests based on either unittest or pytest.\n", "* [Constraints](https://tdda.readthedocs.io/en/tdda-1.0.13/constraints.html) is used to retrieve constraints from a (pandas) DataFrame, write them out as JSON and check whether records satisfy the constraints in the constraints file. It also supports tables in a variety of relational databases.\n", "* [Rexpy](https://tdda.readthedocs.io/en/v1.0.30/rexpy.html) is a tool for automatically deriving regular expressions from a column in a pandas DataFrame or from a (Python) list of examples." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:04.072778Z", "iopub.status.busy": "2026-05-20T11:25:04.072599Z", "iopub.status.idle": "2026-05-20T11:25:04.318018Z", "shell.execute_reply": "2026-05-20T11:25:04.317735Z", "shell.execute_reply.started": "2026-05-20T11:25:04.072760Z" } }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "import pandas as pd\n", "\n", "from tdda.constraints import detect_df, discover_df, verify_df" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:04.318829Z", "iopub.status.busy": "2026-05-20T11:25:04.318490Z", "iopub.status.idle": "2026-05-20T11:25:06.950734Z", "shell.execute_reply": "2026-05-20T11:25:06.950415Z", "shell.execute_reply.started": "2026-05-20T11:25:04.318818Z" } }, "outputs": [], "source": [ "df = pd.read_csv(\n", " \"https://raw.githubusercontent.com/kjam/data-cleaning-101/master/data/iot_example.csv\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Check data\n", "\n", "With [pandas.DataFrame.sample](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html) we display ten random data sets:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:06.951252Z", "iopub.status.busy": "2026-05-20T11:25:06.951161Z", "iopub.status.idle": "2026-05-20T11:25:06.959544Z", "shell.execute_reply": "2026-05-20T11:25:06.959241Z", "shell.execute_reply.started": "2026-05-20T11:25:06.951244Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampusernametemperatureheartratebuildlatestnote
700752017-01-29T12:23:30mary09288684e04441-1d5e-2c42-5f69-88e6695dadd70sleep
961612017-02-08T22:47:44rickycombs236916b49999-ae32-7642-7561-a29cd4c4b4d10NaN
975322017-02-09T11:58:38pamelawilson2775dccf0c7d-2438-9a2b-ed38-970421a8f0720user
605072017-01-25T16:18:01iblankenship11786348eb20-f0b4-7c08-2c86-c48d442eb2ad0sleep
1362392017-02-24T22:27:19nathan512986758ea74b-a8ea-17fd-0b82-4431c593bc3e0NaN
498382017-01-21T10:02:26rachel041878d3ccbe78-60b0-bf3f-c312-83652b2ad7400sleep
1160262017-02-16T20:47:21kimberlytaylor1884ddef2b34-b211-463e-9a74-6a4dfb3270010sleep
304522017-01-13T15:53:55adam5788130625faf-f5b3-44d3-129d-2ddd149c9f691wake
1439522017-02-28T00:40:19breynolds16673f7785f3-3f63-df31-b6c0-97948c1194ee0interval
1382002017-02-25T17:25:19timothyrodriguez208299ff17ba-f7dd-d3ad-1545-ae4392a4bd870user
\n", "
" ], "text/plain": [ " timestamp username temperature heartrate \\\n", "70075 2017-01-29T12:23:30 mary09 28 86 \n", "96161 2017-02-08T22:47:44 rickycombs 23 69 \n", "97532 2017-02-09T11:58:38 pamelawilson 27 75 \n", "60507 2017-01-25T16:18:01 iblankenship 11 78 \n", "136239 2017-02-24T22:27:19 nathan51 29 86 \n", "49838 2017-01-21T10:02:26 rachel04 18 78 \n", "116026 2017-02-16T20:47:21 kimberlytaylor 18 84 \n", "30452 2017-01-13T15:53:55 adam57 8 81 \n", "143952 2017-02-28T00:40:19 breynolds 16 67 \n", "138200 2017-02-25T17:25:19 timothyrodriguez 20 82 \n", "\n", " build latest note \n", "70075 84e04441-1d5e-2c42-5f69-88e6695dadd7 0 sleep \n", "96161 16b49999-ae32-7642-7561-a29cd4c4b4d1 0 NaN \n", "97532 dccf0c7d-2438-9a2b-ed38-970421a8f072 0 user \n", "60507 6348eb20-f0b4-7c08-2c86-c48d442eb2ad 0 sleep \n", "136239 758ea74b-a8ea-17fd-0b82-4431c593bc3e 0 NaN \n", "49838 d3ccbe78-60b0-bf3f-c312-83652b2ad740 0 sleep \n", "116026 ddef2b34-b211-463e-9a74-6a4dfb327001 0 sleep \n", "30452 30625faf-f5b3-44d3-129d-2ddd149c9f69 1 wake \n", "143952 3f7785f3-3f63-df31-b6c0-97948c1194ee 0 interval \n", "138200 99ff17ba-f7dd-d3ad-1545-ae4392a4bd87 0 user " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.sample(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And with [pandas.DataFrame.dtypes](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.dtypes.html) we display the data types for the individual columns:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:06.960138Z", "iopub.status.busy": "2026-05-20T11:25:06.960044Z", "iopub.status.idle": "2026-05-20T11:25:06.962655Z", "shell.execute_reply": "2026-05-20T11:25:06.962428Z", "shell.execute_reply.started": "2026-05-20T11:25:06.960130Z" } }, "outputs": [ { "data": { "text/plain": [ "timestamp object\n", "username object\n", "temperature int64\n", "heartrate int64\n", "build object\n", "latest int64\n", "note object\n", "dtype: object" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Creating a constraints object\n", "\n", "With `discover_constraints` a constraints object can be created." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:06.963097Z", "iopub.status.busy": "2026-05-20T11:25:06.963022Z", "iopub.status.idle": "2026-05-20T11:25:07.170818Z", "shell.execute_reply": "2026-05-20T11:25:07.170480Z", "shell.execute_reply.started": "2026-05-20T11:25:06.963090Z" } }, "outputs": [], "source": [ "constraints = discover_df(df)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:07.172998Z", "iopub.status.busy": "2026-05-20T11:25:07.172845Z", "iopub.status.idle": "2026-05-20T11:25:07.175497Z", "shell.execute_reply": "2026-05-20T11:25:07.175243Z", "shell.execute_reply.started": "2026-05-20T11:25:07.172984Z" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "constraints" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:07.175857Z", "iopub.status.busy": "2026-05-20T11:25:07.175776Z", "iopub.status.idle": "2026-05-20T11:25:07.178899Z", "shell.execute_reply": "2026-05-20T11:25:07.178554Z", "shell.execute_reply.started": "2026-05-20T11:25:07.175848Z" } }, "outputs": [ { "data": { "text/plain": [ "Fields([('timestamp', ),\n", " ('username', ),\n", " ('temperature',\n", " ),\n", " ('heartrate', ),\n", " ('build', ),\n", " ('latest', ),\n", " ('note', )])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "constraints.fields" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Writing the constraints into a file" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:07.179332Z", "iopub.status.busy": "2026-05-20T11:25:07.179246Z", "iopub.status.idle": "2026-05-20T11:25:07.181468Z", "shell.execute_reply": "2026-05-20T11:25:07.181254Z", "shell.execute_reply.started": "2026-05-20T11:25:07.179324Z" } }, "outputs": [], "source": [ "with Path.open(\"../../data/iot_example.json\", \"w\") as f:\n", " f.write(constraints.to_json())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we take a closer look at the file, we can see that, for example, a string with 19 characters is expected for the `timestamp` column and `temperature` expects integers with values from 5-29." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:07.181844Z", "iopub.status.busy": "2026-05-20T11:25:07.181767Z", "iopub.status.idle": "2026-05-20T11:25:07.317302Z", "shell.execute_reply": "2026-05-20T11:25:07.316889Z", "shell.execute_reply.started": "2026-05-20T11:25:07.181837Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"creation_metadata\": {\n", " \"local_time\": \"2026-05-20T13:25:07\",\n", " \"utc_time\": \"2026-05-20T11:25:07\",\n", " \"creator\": \"TDDA 2.2.05\",\n", " \"host\": \"fay.local\",\n", " \"user\": \"veit\",\n", " \"n_records\": 146397,\n", " \"n_selected\": 146397\n", " },\n", " \"fields\": {\n", " \"timestamp\": {\n", " \"type\": \"string\",\n", " \"min_length\": 19,\n", " \"max_length\": 19,\n", " \"max_nulls\": 0,\n", " \"no_duplicates\": true\n", " },\n", " \"username\": {\n", " \"type\": \"string\",\n", " \"min_length\": 3,\n", " \"max_length\": 21,\n", " \"max_nulls\": 0\n", " },\n", " \"temperature\": {\n", " \"type\": \"int\",\n", " \"min\": 5,\n", " \"max\": 29,\n", " \"sign\": \"positive\",\n", " \"max_nulls\": 0\n", " },\n", " \"heartrate\": {\n", " \"type\": \"int\",\n", " \"min\": 60,\n", " \"max\": 89,\n", " \"sign\": \"positive\",\n", " \"max_nulls\": 0\n", " },\n", " \"build\": {\n", " \"type\": \"string\",\n", " \"min_length\": 36,\n", " \"max_length\": 36,\n", " \"max_nulls\": 0,\n", " \"no_duplicates\": true\n", " },\n", " \"latest\": {\n", " \"type\": \"int\",\n", " \"min\": 0,\n", " \"max\": 1,\n", " \"sign\": \"non-negative\",\n", " \"max_nulls\": 0\n", " },\n", " \"note\": {\n", " \"type\": \"string\",\n", " \"min_length\": 4,\n", " \"max_length\": 8,\n", " \"allowed_values\": [\n", " \"interval\",\n", " \"sleep\",\n", " \"test\",\n", " \"update\",\n", " \"user\",\n", " \"wake\"\n", " ]\n", " }\n", " }\n", "}\n" ] } ], "source": [ "!cat ../../data/iot_example.json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Checking data frames\n", "\n", "To do this, we first read in a new csv file with pandas and then have ten data records output as examples:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:07.318148Z", "iopub.status.busy": "2026-05-20T11:25:07.318030Z", "iopub.status.idle": "2026-05-20T11:25:09.567953Z", "shell.execute_reply": "2026-05-20T11:25:09.567637Z", "shell.execute_reply.started": "2026-05-20T11:25:07.318137Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampusernametemperatureheartratebuildlatestnote
438862017-01-19T00:54:05qknight12.0650ced430f-35af-3792-7c5b-0ad5a876f9471.0user
187512017-01-08T23:43:53hrojasNaN88NaN1.0NaN
198392017-01-09T10:08:46thernandezNaN84406e8b71-aa5e-883c-2cc6-5f320f81ec411.0wake
117482017-01-06T04:41:15fdavis28.0799f24681a-754a-f9a2-da4e-9411722675f00.0NaN
309122017-01-13T20:20:59jordandustinNaN8664439e81-1560-73f2-0bea-099af9e979c71.0interval
1024272017-02-11T10:39:43wallsamanthaNaN61NaN0.0wake
647882017-01-27T09:38:51smithshellyNaN6739ec9837-23b5-418e-ab70-580322b3cd10NaNupdate
812752017-02-03T00:09:03alisonhouse26.0671e453b86-b292-81ae-4bdf-a25897ff1234NaNtest
1027432017-02-11T13:41:40beardjohn14.088e003517c-c5bd-c1c5-d326-b9ebd762d2ae1.0test
1047122017-02-12T08:33:52nescobar26.062b705897f-206e-9cfb-4ced-75d7179732e10.0wake
\n", "
" ], "text/plain": [ " timestamp username temperature heartrate \\\n", "43886 2017-01-19T00:54:05 qknight 12.0 65 \n", "18751 2017-01-08T23:43:53 hrojas NaN 88 \n", "19839 2017-01-09T10:08:46 thernandez NaN 84 \n", "11748 2017-01-06T04:41:15 fdavis 28.0 79 \n", "30912 2017-01-13T20:20:59 jordandustin NaN 86 \n", "102427 2017-02-11T10:39:43 wallsamantha NaN 61 \n", "64788 2017-01-27T09:38:51 smithshelly NaN 67 \n", "81275 2017-02-03T00:09:03 alisonhouse 26.0 67 \n", "102743 2017-02-11T13:41:40 beardjohn 14.0 88 \n", "104712 2017-02-12T08:33:52 nescobar 26.0 62 \n", "\n", " build latest note \n", "43886 0ced430f-35af-3792-7c5b-0ad5a876f947 1.0 user \n", "18751 NaN 1.0 NaN \n", "19839 406e8b71-aa5e-883c-2cc6-5f320f81ec41 1.0 wake \n", "11748 9f24681a-754a-f9a2-da4e-9411722675f0 0.0 NaN \n", "30912 64439e81-1560-73f2-0bea-099af9e979c7 1.0 interval \n", "102427 NaN 0.0 wake \n", "64788 39ec9837-23b5-418e-ab70-580322b3cd10 NaN update \n", "81275 1e453b86-b292-81ae-4bdf-a25897ff1234 NaN test \n", "102743 e003517c-c5bd-c1c5-d326-b9ebd762d2ae 1.0 test \n", "104712 b705897f-206e-9cfb-4ced-75d7179732e1 0.0 wake " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_df = pd.read_csv(\n", " \"https://raw.githubusercontent.com/kjam/data-cleaning-101/master/data/iot_example_with_nulls.csv\",\n", ")\n", "new_df.sample(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see several fields that are output as `NaN`. Now, to analyse this systematically, we apply [verify_df](https://tdda.readthedocs.io/en/v1.0.31/constraints.html#tdda.constraints.verify_df) to our new DataFrame. Here, `passes` returns the number of passed constraints, and `failures` returns the number of failed constraints." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.568586Z", "iopub.status.busy": "2026-05-20T11:25:09.568484Z", "iopub.status.idle": "2026-05-20T11:25:09.756275Z", "shell.execute_reply": "2026-05-20T11:25:09.755880Z", "shell.execute_reply.started": "2026-05-20T11:25:09.568577Z" } }, "outputs": [], "source": [ "v = verify_df(new_df, \"../../data/iot_example.json\")" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.756725Z", "iopub.status.busy": "2026-05-20T11:25:09.756649Z", "iopub.status.idle": "2026-05-20T11:25:09.759244Z", "shell.execute_reply": "2026-05-20T11:25:09.758961Z", "shell.execute_reply.started": "2026-05-20T11:25:09.756717Z" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.759618Z", "iopub.status.busy": "2026-05-20T11:25:09.759532Z", "iopub.status.idle": "2026-05-20T11:25:09.761600Z", "shell.execute_reply": "2026-05-20T11:25:09.761351Z", "shell.execute_reply.started": "2026-05-20T11:25:09.759610Z" } }, "outputs": [ { "data": { "text/plain": [ "30" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v.passes" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.762189Z", "iopub.status.busy": "2026-05-20T11:25:09.762075Z", "iopub.status.idle": "2026-05-20T11:25:09.764239Z", "shell.execute_reply": "2026-05-20T11:25:09.764032Z", "shell.execute_reply.started": "2026-05-20T11:25:09.762177Z" } }, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v.failures" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also display which constraints passed and failed in which columns:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.765004Z", "iopub.status.busy": "2026-05-20T11:25:09.764830Z", "iopub.status.idle": "2026-05-20T11:25:09.766708Z", "shell.execute_reply": "2026-05-20T11:25:09.766389Z", "shell.execute_reply.started": "2026-05-20T11:25:09.764994Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "FIELDS:\n", "\n", "timestamp: 0 failures 5 passes type ✓ min_length ✓ max_length ✓ max_nulls ✓ no_duplicates ✓\n", "\n", "username: 0 failures 4 passes type ✓ min_length ✓ max_length ✓ max_nulls ✓\n", "\n", "temperature: 1 failure 4 passes type ✓ min ✓ max ✓ sign ✓ max_nulls ✗\n", "\n", "heartrate: 0 failures 5 passes type ✓ min ✓ max ✓ sign ✓ max_nulls ✓\n", "\n", "build: 1 failure 4 passes type ✓ min_length ✓ max_length ✓ max_nulls ✗ no_duplicates ✓\n", "\n", "latest: 1 failure 4 passes type ✓ min ✓ max ✓ sign ✓ max_nulls ✗\n", "\n", "note: 0 failures 4 passes type ✓ min_length ✓ max_length ✓ allowed_values ✓\n", "\n", "SUMMARY:\n", "\n", "Constraints passing: 30\n", "Constraints failing: 3\n" ] } ], "source": [ "print(str(v))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternatively, we can also display these results in tabular form:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.767256Z", "iopub.status.busy": "2026-05-20T11:25:09.767153Z", "iopub.status.idle": "2026-05-20T11:25:09.773077Z", "shell.execute_reply": "2026-05-20T11:25:09.772777Z", "shell.execute_reply.started": "2026-05-20T11:25:09.767244Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
fieldfailurespassestypeminmin_lengthmaxmax_lengthsignmax_nullsno_duplicatesallowed_values
0timestamp05TrueNaNTrueNaNTrueNaNTrueTrueNaN
1username04TrueNaNTrueNaNTrueNaNTrueNaNNaN
2temperature14TrueTrueNaNTrueNaNTrueFalseNaNNaN
3heartrate05TrueTrueNaNTrueNaNTrueTrueNaNNaN
4build14TrueNaNTrueNaNTrueNaNFalseTrueNaN
5latest14TrueTrueNaNTrueNaNTrueFalseNaNNaN
6note04TrueNaNTrueNaNTrueNaNNaNNaNTrue
\n", "
" ], "text/plain": [ " field failures passes type min min_length max max_length \\\n", "0 timestamp 0 5 True NaN True NaN True \n", "1 username 0 4 True NaN True NaN True \n", "2 temperature 1 4 True True NaN True NaN \n", "3 heartrate 0 5 True True NaN True NaN \n", "4 build 1 4 True NaN True NaN True \n", "5 latest 1 4 True True NaN True NaN \n", "6 note 0 4 True NaN True NaN True \n", "\n", " sign max_nulls no_duplicates allowed_values \n", "0 NaN True True NaN \n", "1 NaN True NaN NaN \n", "2 True False NaN NaN \n", "3 True True NaN NaN \n", "4 NaN False True NaN \n", "5 True False NaN NaN \n", "6 NaN NaN NaN True " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "v.to_frame()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Finding the faulty rows\n", "\n", "`tdda.constraints.pd.constraints.detect_df()` detects records in the pandas DataFrame that violate one of the constraints in the provided JSON file. We can then call the `detected()` function on the created `PandasDetection` object to output the rows that are faulty:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.773767Z", "iopub.status.busy": "2026-05-20T11:25:09.773687Z", "iopub.status.idle": "2026-05-20T11:25:09.969876Z", "shell.execute_reply": "2026-05-20T11:25:09.969605Z", "shell.execute_reply.started": "2026-05-20T11:25:09.773758Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
n_failures
Index
31
41
71
102
121
......
1463851
1463872
1463912
1463932
1463941
\n", "

77260 rows × 1 columns

\n", "
" ], "text/plain": [ " n_failures\n", "Index \n", "3 1\n", "4 1\n", "7 1\n", "10 2\n", "12 1\n", "... ...\n", "146385 1\n", "146387 2\n", "146391 2\n", "146393 2\n", "146394 1\n", "\n", "[77260 rows x 1 columns]" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = detect_df(new_df, \"iot_example.json\")\n", "\n", "d.detected()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can display all incorrect data records by using only the part of the index from `new_df` that also appears in `d.detected()`:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.970492Z", "iopub.status.busy": "2026-05-20T11:25:09.970309Z", "iopub.status.idle": "2026-05-20T11:25:09.972653Z", "shell.execute_reply": "2026-05-20T11:25:09.972409Z", "shell.execute_reply.started": "2026-05-20T11:25:09.970479Z" } }, "outputs": [], "source": [ "d_index = d.detected().index" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.973086Z", "iopub.status.busy": "2026-05-20T11:25:09.972975Z", "iopub.status.idle": "2026-05-20T11:25:09.980867Z", "shell.execute_reply": "2026-05-20T11:25:09.980584Z", "shell.execute_reply.started": "2026-05-20T11:25:09.973076Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampusernametemperatureheartratebuildlatestnote
32017-01-01T12:02:09eddierodriguez28.076NaN0.0update
42017-01-01T12:02:36kenneth9429.062122f1c6a-403c-2221-6ed1-b5caa08f11e0NaNNaN
72017-01-01T12:04:35scott2816.0767a60219f-6621-e548-180e-ca69624f9824NaNinterval
102017-01-01T12:06:21njohnsonNaN63e09b6001-125d-51cf-9c3f-9cb686c19d02NaNNaN
122017-01-01T12:07:41jessica4822.08303e1a07b-3e14-412c-3a69-6b45bc79f81cNaNupdate
........................
1463852017-02-28T23:53:59powelleric20.086152eda10-676a-069c-b664-19443f2c8081NaNtest
1463872017-02-28T23:54:50jthompsonNaN668da10303-fe49-e313-8fda-0d5e79ded054NaNupdate
1463912017-02-28T23:57:21aaronbeckerNaN877e52f4a8-345c-5ee0-e515-b8c392213062NaNsleep
1463932017-02-28T23:58:43joelrussoNaN89NaN0.0NaN
1463942017-02-28T23:59:23lellisNaN84dac87426-e147-9c39-6e4c-790bb11f8fc90.0update
\n", "

77260 rows × 7 columns

\n", "
" ], "text/plain": [ " timestamp username temperature heartrate \\\n", "3 2017-01-01T12:02:09 eddierodriguez 28.0 76 \n", "4 2017-01-01T12:02:36 kenneth94 29.0 62 \n", "7 2017-01-01T12:04:35 scott28 16.0 76 \n", "10 2017-01-01T12:06:21 njohnson NaN 63 \n", "12 2017-01-01T12:07:41 jessica48 22.0 83 \n", "... ... ... ... ... \n", "146385 2017-02-28T23:53:59 powelleric 20.0 86 \n", "146387 2017-02-28T23:54:50 jthompson NaN 66 \n", "146391 2017-02-28T23:57:21 aaronbecker NaN 87 \n", "146393 2017-02-28T23:58:43 joelrusso NaN 89 \n", "146394 2017-02-28T23:59:23 lellis NaN 84 \n", "\n", " build latest note \n", "3 NaN 0.0 update \n", "4 122f1c6a-403c-2221-6ed1-b5caa08f11e0 NaN NaN \n", "7 7a60219f-6621-e548-180e-ca69624f9824 NaN interval \n", "10 e09b6001-125d-51cf-9c3f-9cb686c19d02 NaN NaN \n", "12 03e1a07b-3e14-412c-3a69-6b45bc79f81c NaN update \n", "... ... ... ... \n", "146385 152eda10-676a-069c-b664-19443f2c8081 NaN test \n", "146387 8da10303-fe49-e313-8fda-0d5e79ded054 NaN update \n", "146391 7e52f4a8-345c-5ee0-e515-b8c392213062 NaN sleep \n", "146393 NaN 0.0 NaN \n", "146394 dac87426-e147-9c39-6e4c-790bb11f8fc9 0.0 update \n", "\n", "[77260 rows x 7 columns]" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_df[new_df.index.isin(d_index)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2026-05-20T11:25:09.981427Z", "iopub.status.busy": "2026-05-20T11:25:09.981314Z", "iopub.status.idle": "2026-05-20T11:25:09.989028Z", "shell.execute_reply": "2026-05-20T11:25:09.988714Z", "shell.execute_reply.started": "2026-05-20T11:25:09.981415Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampusernametemperatureheartratebuildlatestnote
02017-01-01T12:00:23michaelsmith12.0674e6a7805-8faa-2768-6ef6-eb3198b483ac0.0interval
12017-01-01T12:01:09kharrison6.0787256b7b0-e502-f576-62ec-ed73533c9c840.0wake
22017-01-01T12:01:34smithadam5.0899226c94b-bb4b-a6c8-8e02-cb42b53e9c900.0NaN
52017-01-01T12:03:04bryanttodd13.0860897dbe5-9c5b-71ca-73a1-7586959ca1980.0interval
62017-01-01T12:03:51andrea9817.0811c07ab9b-5f66-137d-a74f-921a41001f4e1.0NaN
........................
1463892017-02-28T23:56:05kathy635.088c2f76050-abd4-aee4-7bc0-3498325d05730.0NaN
1463902017-02-28T23:56:34cookallison16.084f0b0c1f9-900b-276c-bca9-ac4d4ec4e88e0.0user
1463922017-02-28T23:58:06mcontreras15.06369e61a15-d2d0-47a7-1a27-e07b3eeeba100.0NaN
1463952017-02-28T23:59:48grayjasmin17.0644911a589-3a15-4bbf-1de1-e5a69ab739da1.0update
1463962017-03-01T00:00:30jgreene23.0704f95bbca-26a7-29e7-1f19-aaedf1a517410.0interval
\n", "

69137 rows × 7 columns

\n", "
" ], "text/plain": [ " timestamp username temperature heartrate \\\n", "0 2017-01-01T12:00:23 michaelsmith 12.0 67 \n", "1 2017-01-01T12:01:09 kharrison 6.0 78 \n", "2 2017-01-01T12:01:34 smithadam 5.0 89 \n", "5 2017-01-01T12:03:04 bryanttodd 13.0 86 \n", "6 2017-01-01T12:03:51 andrea98 17.0 81 \n", "... ... ... ... ... \n", "146389 2017-02-28T23:56:05 kathy63 5.0 88 \n", "146390 2017-02-28T23:56:34 cookallison 16.0 84 \n", "146392 2017-02-28T23:58:06 mcontreras 15.0 63 \n", "146395 2017-02-28T23:59:48 grayjasmin 17.0 64 \n", "146396 2017-03-01T00:00:30 jgreene 23.0 70 \n", "\n", " build latest note \n", "0 4e6a7805-8faa-2768-6ef6-eb3198b483ac 0.0 interval \n", "1 7256b7b0-e502-f576-62ec-ed73533c9c84 0.0 wake \n", "2 9226c94b-bb4b-a6c8-8e02-cb42b53e9c90 0.0 NaN \n", "5 0897dbe5-9c5b-71ca-73a1-7586959ca198 0.0 interval \n", "6 1c07ab9b-5f66-137d-a74f-921a41001f4e 1.0 NaN \n", "... ... ... ... \n", "146389 c2f76050-abd4-aee4-7bc0-3498325d0573 0.0 NaN \n", "146390 f0b0c1f9-900b-276c-bca9-ac4d4ec4e88e 0.0 user \n", "146392 69e61a15-d2d0-47a7-1a27-e07b3eeeba10 0.0 NaN \n", "146395 4911a589-3a15-4bbf-1de1-e5a69ab739da 1.0 update \n", "146396 4f95bbca-26a7-29e7-1f19-aaedf1a51741 0.0 interval \n", "\n", "[69137 rows x 7 columns]" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_df[~new_df.index.isin(d_index)]" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.13 Kernel", "language": "python", "name": "python313" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.0" }, "latex_envs": { "LaTeX_envs_menu_present": true, "autoclose": false, "autocomplete": true, "bibliofile": "biblio.bib", "cite_by": "apalike", "current_citInitial": 1, "eqLabelWithNumbers": true, "eqNumInitial": 1, "hotkeys": { "equation": "Ctrl-E", "itemize": "Ctrl-I" }, "labels_anchors": false, "latex_user_defs": false, "report_style_numbering": false, "user_envs_cfg": false }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }