Categories of data quality

There are various ways to categorise data quality issues – by the cause of the problem, by its likely or potential impact, by the type of data element, or by the level of a data hierarchy at which the error or inconsistency becomes apparent, to name but a few. In this tutorial, categorisation is generally implicit, and we focus on the methods for detecting problems and the tools suitable for this purpose. However, it is useful to at least briefly touch upon other ways of categorising data problems before we turn our attention to detection.

Impossible data

such as temperatures below absolute zero (0 Kelvin or −273.15 °C) or nonsensical date and time values – whilst these cannot occur in the datetime data type in Python, they can occur in JSON and SQLite, where text fields are usually used to store date information.

Special values

The IEEE-754 standard for representing floating-point numbers also includes special values for positive and negative infinity (+∞, −∞) as well as Not a Number (NaN) values.

Unspecified Values

Values that conform to a specific format may nevertheless be invalid, such as email addresses.

Inconsistencies

Groups of related field values for each record may be mutually exclusive. To avoid such inconsistencies, the DRY principle recommends disregarding calculable values. Checksums, however, are the exact opposite of this.

‘Single Source of Truth’ is an example of database normalisation and simplifies updates; with denormalised data stores, on the other hand, joins and calculations can be avoided during data analysis.

Anomalies and data drift

The checks carried out to date have made it possible to declare data invalid. However, anomalies and data drift do not necessarily support this conclusion and therefore rarely allow for automated data cleansing.

Malware

Not all input or output data should be processed in analysis systems.

Exploits of a Mom

Ein offensichtliches Beispiel für versehentlich offengelegte Ausgabedaten sind z. B. aus fortlaufenden Personenkennziffern generierte URLs mit personenbezogenen Daten.