Intake for data scientists

Intake makes it easy to load many different formats and types. For a complete overview, take a look at the Plugin Directory and the Intake Project Dashboard. Intake then transfers the data to common storage formats such as Pandas DataFrames, Numpy arrays or Python lists. They are then easily searchable and also accessible to distributed systems. If you are missing a plugin, you can also order one yourself, as described in Making Drivers.

Load a data source

Hereinafter we will read two csv data records and transfer them to an intake catalog.

[1]:
import intake


ds = intake.open_csv("states_*.csv")

print(ds)
sources:
  csv:
    args:
      urlpath: states_*.csv
    description: ''
    driver: intake.source.csv.CSVSource
    metadata: {}

Mit der open_*-Funktion von Intake lassen sich verschiedenen Datenquellen einlesen. Je nach Datenformat oder Dienst lassen sich unterschiedliche Argmuente verwenden.

Configure the search path for data sources

Intake checks the Intake configuration file for catalog_path and the environment variable "INTAKE_PATH" for a colon-separated list of paths or semicolons in Windows to look for catalog files. When importing intake, all entries from all catalogs that are referenced by intake.cat as part of a global catalog are displayed.

Read data

Intake reads data in containers of various formats:

  • Tables in Pandas DataFrames

  • Multi-dimensional arrays in numpy arrays

  • Semi-structured data in Python lists of objects, usually dictionaries

To find out in which container format Intake holds the data, you can use the container attribute:

[2]:
ds.container
[2]:
'dataframe'

In addition to dataframe, the result can also be ndarray or python.

[3]:
df = ds.read()

df.head()
[3]:
state slug code nickname website admission_date admission_number capital_city capital_url population population_rank constitution_url state_flag_url state_seal_url map_image_url landscape_background_url skyline_background_url twitter_url facebook_url
0 Alabama alabama AL Yellowhammer State http://www.alabama.gov 1819-12-14 22 Montgomery http://www.montgomeryal.gov 4833722 23 http://alisondb.legislature.state.al.us/alison... https://cdn.civil.services/us-states/flags/ala... https://cdn.civil.services/us-states/seals/ala... https://cdn.civil.services/us-states/maps/alab... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/alabamagov https://www.facebook.com/alabamagov
1 Alaska alaska AK The Last Frontier http://alaska.gov 1959-01-03 49 Juneau http://www.juneau.org 735132 47 http://www.legis.state.ak.us/basis/folioproxy.... https://cdn.civil.services/us-states/flags/ala... https://cdn.civil.services/us-states/seals/ala... https://cdn.civil.services/us-states/maps/alas... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/alaska https://www.facebook.com/AlaskaLocalGovernments
2 Arizona arizona AZ The Grand Canyon State https://az.gov 1912-02-14 48 Phoenix https://www.phoenix.gov 6626624 15 http://www.azleg.gov/Constitution.asp https://cdn.civil.services/us-states/flags/ari... https://cdn.civil.services/us-states/seals/ari... https://cdn.civil.services/us-states/maps/ariz... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... NaN NaN
3 Arkansas arkansas AR The Natural State http://arkansas.gov 1836-06-15 25 Little Rock http://www.littlerock.org 2959373 32 http://www.arkleg.state.ar.us/assembly/Summary... https://cdn.civil.services/us-states/flags/ark... https://cdn.civil.services/us-states/seals/ark... https://cdn.civil.services/us-states/maps/arka... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/arkansasgov https://www.facebook.com/Arkansas.gov
4 California california CA Golden State http://www.ca.gov 1850-09-09 31 Sacramento http://www.cityofsacramento.org 38332521 1 http://www.leginfo.ca.gov/const-toc.html https://cdn.civil.services/us-states/flags/cal... https://cdn.civil.services/us-states/seals/cal... https://cdn.civil.services/us-states/maps/cali... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/cagovernment NaN
[4]:
for chunk in ds.read_chunked():
    print("Chunk: %d" % len(chunk))
Chunk: 24
Chunk: 26
[5]:
ddf = ds.to_dask()

ddf.head()
[5]:
state slug code nickname website admission_date admission_number capital_city capital_url population population_rank constitution_url state_flag_url state_seal_url map_image_url landscape_background_url skyline_background_url twitter_url facebook_url
0 Alabama alabama AL Yellowhammer State http://www.alabama.gov 1819-12-14 22 Montgomery http://www.montgomeryal.gov 4833722 23 http://alisondb.legislature.state.al.us/alison... https://cdn.civil.services/us-states/flags/ala... https://cdn.civil.services/us-states/seals/ala... https://cdn.civil.services/us-states/maps/alab... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/alabamagov https://www.facebook.com/alabamagov
1 Alaska alaska AK The Last Frontier http://alaska.gov 1959-01-03 49 Juneau http://www.juneau.org 735132 47 http://www.legis.state.ak.us/basis/folioproxy.... https://cdn.civil.services/us-states/flags/ala... https://cdn.civil.services/us-states/seals/ala... https://cdn.civil.services/us-states/maps/alas... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/alaska https://www.facebook.com/AlaskaLocalGovernments
2 Arizona arizona AZ The Grand Canyon State https://az.gov 1912-02-14 48 Phoenix https://www.phoenix.gov 6626624 15 http://www.azleg.gov/Constitution.asp https://cdn.civil.services/us-states/flags/ari... https://cdn.civil.services/us-states/seals/ari... https://cdn.civil.services/us-states/maps/ariz... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... <NA> <NA>
3 Arkansas arkansas AR The Natural State http://arkansas.gov 1836-06-15 25 Little Rock http://www.littlerock.org 2959373 32 http://www.arkleg.state.ar.us/assembly/Summary... https://cdn.civil.services/us-states/flags/ark... https://cdn.civil.services/us-states/seals/ark... https://cdn.civil.services/us-states/maps/arka... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/arkansasgov https://www.facebook.com/Arkansas.gov
4 California california CA Golden State http://www.ca.gov 1850-09-09 31 Sacramento http://www.cityofsacramento.org 38332521 1 http://www.leginfo.ca.gov/const-toc.html https://cdn.civil.services/us-states/flags/cal... https://cdn.civil.services/us-states/seals/cal... https://cdn.civil.services/us-states/maps/cali... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/cagovernment <NA>
[6]:
cat = intake.open_catalog("us_states.yml")
[7]:
list(cat)
[7]:
['states']
[8]:
cat.states.to_dask()[["state", "slug"]].head()
[8]:
state slug
0 Alabama alabama
1 Alaska alaska
2 Arizona arizona
3 Arkansas arkansas
4 California california
[9]:
cat.states(csv_kwargs={"header": None, "skiprows": 1}).read().head()
[9]:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
0 Alabama alabama AL Yellowhammer State http://www.alabama.gov 1819-12-14 22 Montgomery http://www.montgomeryal.gov 4833722 23 http://alisondb.legislature.state.al.us/alison... https://cdn.civil.services/us-states/flags/ala... https://cdn.civil.services/us-states/seals/ala... https://cdn.civil.services/us-states/maps/alab... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/alabamagov https://www.facebook.com/alabamagov
1 Alaska alaska AK The Last Frontier http://alaska.gov 1959-01-03 49 Juneau http://www.juneau.org 735132 47 http://www.legis.state.ak.us/basis/folioproxy.... https://cdn.civil.services/us-states/flags/ala... https://cdn.civil.services/us-states/seals/ala... https://cdn.civil.services/us-states/maps/alas... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/alaska https://www.facebook.com/AlaskaLocalGovernments
2 Arizona arizona AZ The Grand Canyon State https://az.gov 1912-02-14 48 Phoenix https://www.phoenix.gov 6626624 15 http://www.azleg.gov/Constitution.asp https://cdn.civil.services/us-states/flags/ari... https://cdn.civil.services/us-states/seals/ari... https://cdn.civil.services/us-states/maps/ariz... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... NaN NaN
3 Arkansas arkansas AR The Natural State http://arkansas.gov 1836-06-15 25 Little Rock http://www.littlerock.org 2959373 32 http://www.arkleg.state.ar.us/assembly/Summary... https://cdn.civil.services/us-states/flags/ark... https://cdn.civil.services/us-states/seals/ark... https://cdn.civil.services/us-states/maps/arka... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/arkansasgov https://www.facebook.com/Arkansas.gov
4 California california CA Golden State http://www.ca.gov 1850-09-09 31 Sacramento http://www.cityofsacramento.org 38332521 1 http://www.leginfo.ca.gov/const-toc.html https://cdn.civil.services/us-states/flags/cal... https://cdn.civil.services/us-states/seals/cal... https://cdn.civil.services/us-states/maps/cali... https://cdn.civil.services/us-states/backgroun... https://cdn.civil.services/us-states/backgroun... https://twitter.com/cagovernment NaN