Intake for data scientists#
Intake makes it easy to load many different formats and types. For a complete overview, take a look at the Plugin Directory and the Intake Project Dashboard. Intake then transfers the data to common storage formats such as Pandas DataFrames, Numpy arrays or Python lists. They are then easily searchable and also accessible to distributed systems. If you are missing a plugin, you can also order one yourself, as described in Making Drivers.
Load a data source#
Hereinafter we will read two csv data records and transfer them to an intake catalog.
[1]:
import intake
ds = intake.open_csv("states_*.csv")
print(ds)
sources:
csv:
args:
urlpath: states_*.csv
description: ''
driver: intake.source.csv.CSVSource
metadata: {}
Mit der open_*-Funktion von Intake lassen sich verschiedenen Datenquellen einlesen. Je nach Datenformat oder Dienst lassen sich unterschiedliche Argmuente verwenden.
Configure the search path for data sources#
Intake checks the Intake configuration file for catalog_path and the environment variable "INTAKE_PATH" for a colon-separated list of paths or semicolons in Windows to look for catalog files. When importing intake, all entries from all catalogs that are referenced by intake.cat as part of a global catalog are displayed.
Read data#
Intake reads data in containers of various formats:
Tables in Pandas DataFrames
Multi-dimensional arrays in numpy arrays
Semi-structured data in Python lists of objects, usually dictionaries
To find out in which container format Intake holds the data, you can use the container attribute:
[2]:
ds.container
[2]:
'dataframe'
In addition to dataframe, the result can also be ndarray or python.
[3]:
df = ds.read()
df.head()
[3]:
| state | slug | code | nickname | website | admission_date | admission_number | capital_city | capital_url | population | population_rank | constitution_url | state_flag_url | state_seal_url | map_image_url | landscape_background_url | skyline_background_url | twitter_url | facebook_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Alabama | alabama | AL | Yellowhammer State | http://www.alabama.gov | 1819-12-14 | 22 | Montgomery | http://www.montgomeryal.gov | 4833722 | 23 | http://alisondb.legislature.state.al.us/alison... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alab... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alabamagov | https://www.facebook.com/alabamagov |
| 1 | Alaska | alaska | AK | The Last Frontier | http://alaska.gov | 1959-01-03 | 49 | Juneau | http://www.juneau.org | 735132 | 47 | http://www.legis.state.ak.us/basis/folioproxy.... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alas... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alaska | https://www.facebook.com/AlaskaLocalGovernments |
| 2 | Arizona | arizona | AZ | The Grand Canyon State | https://az.gov | 1912-02-14 | 48 | Phoenix | https://www.phoenix.gov | 6626624 | 15 | http://www.azleg.gov/Constitution.asp | https://cdn.civil.services/us-states/flags/ari... | https://cdn.civil.services/us-states/seals/ari... | https://cdn.civil.services/us-states/maps/ariz... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | NaN | NaN |
| 3 | Arkansas | arkansas | AR | The Natural State | http://arkansas.gov | 1836-06-15 | 25 | Little Rock | http://www.littlerock.org | 2959373 | 32 | http://www.arkleg.state.ar.us/assembly/Summary... | https://cdn.civil.services/us-states/flags/ark... | https://cdn.civil.services/us-states/seals/ark... | https://cdn.civil.services/us-states/maps/arka... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/arkansasgov | https://www.facebook.com/Arkansas.gov |
| 4 | California | california | CA | Golden State | http://www.ca.gov | 1850-09-09 | 31 | Sacramento | http://www.cityofsacramento.org | 38332521 | 1 | http://www.leginfo.ca.gov/const-toc.html | https://cdn.civil.services/us-states/flags/cal... | https://cdn.civil.services/us-states/seals/cal... | https://cdn.civil.services/us-states/maps/cali... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/cagovernment | NaN |
[4]:
for chunk in ds.read_chunked():
print("Chunk: %d" % len(chunk))
Chunk: 24
Chunk: 26
[5]:
ddf = ds.to_dask()
ddf.head()
[5]:
| state | slug | code | nickname | website | admission_date | admission_number | capital_city | capital_url | population | population_rank | constitution_url | state_flag_url | state_seal_url | map_image_url | landscape_background_url | skyline_background_url | twitter_url | facebook_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Alabama | alabama | AL | Yellowhammer State | http://www.alabama.gov | 1819-12-14 | 22 | Montgomery | http://www.montgomeryal.gov | 4833722 | 23 | http://alisondb.legislature.state.al.us/alison... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alab... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alabamagov | https://www.facebook.com/alabamagov |
| 1 | Alaska | alaska | AK | The Last Frontier | http://alaska.gov | 1959-01-03 | 49 | Juneau | http://www.juneau.org | 735132 | 47 | http://www.legis.state.ak.us/basis/folioproxy.... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alas... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alaska | https://www.facebook.com/AlaskaLocalGovernments |
| 2 | Arizona | arizona | AZ | The Grand Canyon State | https://az.gov | 1912-02-14 | 48 | Phoenix | https://www.phoenix.gov | 6626624 | 15 | http://www.azleg.gov/Constitution.asp | https://cdn.civil.services/us-states/flags/ari... | https://cdn.civil.services/us-states/seals/ari... | https://cdn.civil.services/us-states/maps/ariz... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | <NA> | <NA> |
| 3 | Arkansas | arkansas | AR | The Natural State | http://arkansas.gov | 1836-06-15 | 25 | Little Rock | http://www.littlerock.org | 2959373 | 32 | http://www.arkleg.state.ar.us/assembly/Summary... | https://cdn.civil.services/us-states/flags/ark... | https://cdn.civil.services/us-states/seals/ark... | https://cdn.civil.services/us-states/maps/arka... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/arkansasgov | https://www.facebook.com/Arkansas.gov |
| 4 | California | california | CA | Golden State | http://www.ca.gov | 1850-09-09 | 31 | Sacramento | http://www.cityofsacramento.org | 38332521 | 1 | http://www.leginfo.ca.gov/const-toc.html | https://cdn.civil.services/us-states/flags/cal... | https://cdn.civil.services/us-states/seals/cal... | https://cdn.civil.services/us-states/maps/cali... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/cagovernment | <NA> |
[6]:
cat = intake.open_catalog("us_states.yml")
[7]:
list(cat)
[7]:
['states']
[8]:
cat.states.to_dask()[["state", "slug"]].head()
[8]:
| state | slug | |
|---|---|---|
| 0 | Alabama | alabama |
| 1 | Alaska | alaska |
| 2 | Arizona | arizona |
| 3 | Arkansas | arkansas |
| 4 | California | california |
[9]:
cat.states(csv_kwargs={"header": None, "skiprows": 1}).read().head()
[9]:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Alabama | alabama | AL | Yellowhammer State | http://www.alabama.gov | 1819-12-14 | 22 | Montgomery | http://www.montgomeryal.gov | 4833722 | 23 | http://alisondb.legislature.state.al.us/alison... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alab... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alabamagov | https://www.facebook.com/alabamagov |
| 1 | Alaska | alaska | AK | The Last Frontier | http://alaska.gov | 1959-01-03 | 49 | Juneau | http://www.juneau.org | 735132 | 47 | http://www.legis.state.ak.us/basis/folioproxy.... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alas... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alaska | https://www.facebook.com/AlaskaLocalGovernments |
| 2 | Arizona | arizona | AZ | The Grand Canyon State | https://az.gov | 1912-02-14 | 48 | Phoenix | https://www.phoenix.gov | 6626624 | 15 | http://www.azleg.gov/Constitution.asp | https://cdn.civil.services/us-states/flags/ari... | https://cdn.civil.services/us-states/seals/ari... | https://cdn.civil.services/us-states/maps/ariz... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | NaN | NaN |
| 3 | Arkansas | arkansas | AR | The Natural State | http://arkansas.gov | 1836-06-15 | 25 | Little Rock | http://www.littlerock.org | 2959373 | 32 | http://www.arkleg.state.ar.us/assembly/Summary... | https://cdn.civil.services/us-states/flags/ark... | https://cdn.civil.services/us-states/seals/ark... | https://cdn.civil.services/us-states/maps/arka... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/arkansasgov | https://www.facebook.com/Arkansas.gov |
| 4 | California | california | CA | Golden State | http://www.ca.gov | 1850-09-09 | 31 | Sacramento | http://www.cityofsacramento.org | 38332521 | 1 | http://www.leginfo.ca.gov/const-toc.html | https://cdn.civil.services/us-states/flags/cal... | https://cdn.civil.services/us-states/seals/cal... | https://cdn.civil.services/us-states/maps/cali... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/cagovernment | NaN |