Import notebooks

To be able to develop more modularly, the import of notebooks is necessary. However, since notebooks are not Python files, they are not easy to import. Fortunately, Python provides some hooks for the import so that IPython notebooks can eventually be imported.

import os
import sys
import types

import nbformat

from IPython import get_ipython
from IPython.core.interactiveshell import InteractiveShell

Import hooks usually have two objects:

• Module Loader that takes a module name (e.g. IPython.display) and returns a module

• Module Finder, which finds out if a module is present and tells Python which loader to use

But first, let’s write a method that a notebook will find using the fully qualified name and the optional path. E.g. mypackage.foo becomes mypackage/foo.ipynb and replaces Foo_Bar with Foo Bar if Foo_Bar doesn’t exist.

def find_notebook(fullname, path=None):
    name = fullname.rsplit(".", 1)[-1]
    if not path:
        path = [""]
    for d in path:
        nb_path = os.path.join(d, name + ".ipynb")
        if os.path.isfile(nb_path):
            return nb_path
        # let import Foo_Bar find "Foo Bar.ipynb"
        nb_path = nb_path.replace("_", " ")
        if os.path.isfile(nb_path):
            return nb_path

Notebook Loader

The Notebook Loader does the following three steps:

1. Load the notebook document into memory

2. Create an empty module

3. Execute every cell in the module namespace

   Because IPython cells can have an extended syntax, transform_cell converts each cell to pure Python code before executing it.

class NotebookLoader(object):
    """Module Loader for IPython Notebooks"""

    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path

    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)

        print("importing notebook from %s" % path)

        # load the notebook object
        nb = nbformat.read(path, as_version=4)

        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        mod.__dict__["get_ipython"] = get_ipython
        sys.modules[fullname] = mod

        # extra work to ensure that magics that would affect the user_ns
        # magics that would affect the user_ns actually affect the
        # notebook module’s ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__

        try:
            for cell in nb.cells:
                if cell.cell_type == "code":
                    # transform the input to executable Python
                    code = self.shell.input_transformer_manager.transform_cell(
                        cell.source
                    )
                    # run the code in the module
                    exec(code, mod.__dict__)
        finally:
            self.shell.user_ns = save_user_ns
        return mod

Notebook Finder

The Finder is a simple object that indicates whether a notebook can be imported based on its file name and that returns the appropriate loader.

class NotebookFinder(object):
    """Module Finder finds the transformed IPython Notebook"""

    def __init__(self):
        self.loaders = {}

    def find_module(self, fullname, path=None):
        nb_path = find_notebook(fullname, path)
        if not nb_path:
            return

        key = path
        if path:
            # lists aren’t hashable
            key = os.path.sep.join(path)

        if key not in self.loaders:
            self.loaders[key] = NotebookLoader(path)
        return self.loaders[key]

Register hook

Now we register NotebookFinder with sys.meta_path:

sys.meta_path.append(NotebookFinder())

Check

Now our notebook mypackage/foo.ipynb should be importable with:

from mypackage import foo

importing notebook from /Users/veit/cusy/trn/Python4DataScience/docs/workspace/ipython/mypackage/foo.ipynb

Is the Python method bar being executed?

foo.bar()

'bar'

… and the IPython syntax?

foo.dirlist()

['debugging.ipynb',
 'display.ipynb',
 'examples.ipynb',
 'extensions.rst',
 'importing.ipynb',
 'index.rst',
 'magics.ipynb',
 '\x1b[34mmypackage\x1b[m\x1b[m',
 'myscript.py',
 'shell.ipynb',
 'start.rst',
 '\x1b[31mtab-completion-for-anything.png\x1b[m\x1b[m',
 '\x1b[31mtab-completion-for-modules.png\x1b[m\x1b[m',
 '\x1b[31mtab-completion-for-objects.png\x1b[m\x1b[m',
 '\x1b[34munix-shell\x1b[m\x1b[m']

Reusable import hook

The import hook can also easily be executed in other notebooks with

%run display.ipynb

• foo
• bar

markdown cell

# `foo.ipynb`

code cell

def bar():
    return "bar"

code cell

def dirlist():
    listing = !ls
    return listing

code cell

def whatsmyname():
    return __name__