IPython magic

IPython not only enables Python to be used interactively, but also extends the Python syntax with so-called magic commands, which are provided with the prefix %. They are designed to quickly and easily solve common data analysis problems. A distinction is made between two different types of magic commands:

• line magics, denoted by a single % prefix, that run on a single input line

• cell magics which are preceded by a double symbol %% and which are executed within a notebook cell.

Execute external code: %run

If you start developing larger code, you will likely be working in both IPython for interactive exploration and a text editor to save code that you want to reuse. With the %run magic you can execute this code directly in your IPython session.

Imagine you created a myscript.py file with the following content:

def square(x):
    return x**2


for N in range(1, 4):
    print(N, "squared is", square(N))

%run myscript.py

1 squared is 1
2 squared is 4
3 squared is 9

Note that after running this script, all of the functions defined in it will be available for use in your IPython session:

square(4)

16

There are several ways you can improve the way your code runs. As usual, you can display the documentation in IPython with %run?.

Run timing code: %timeit

Another example of a Magic function is %timeit, which automatically determines the execution time of the following one-line Python statement. So we can e.g. output the performance of a list comprehension with:

%timeit L = [n ** 2 for n in range(1000)]

27.6 µs ± 290 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

The advantage of %timeit is that short commands automatically run multiple runs to get more robust results. For multi-line instructions, adding a second % character creates cell magic that can process multiple input lines. For example, here is the equivalent construction using a for loop:

%%timeit
L = []
for n in range(1000):
    L.append(n ** 2)

29.7 µs ± 207 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

We can immediately see that the list comprehension is about 10% faster than its equivalent with a for loop. We then describe performance measurements and optimisations in more detail in Profiling.

Execute code from other interpreters

IPython has a %%script script magic with which you can execute a cell in a subprocess of an interpreter on your system, e.g. bash, ruby, perl, zsh, R etc. This can also be its own script that expects input in stdin. To do this, simply pass a path or a shell command to the program that is specified in the %%script line. The rest of the cell is executed by this script, capturing stdout or err from the subprocess and displaying it.

%%script python2
import sys


print("Python: %s" % sys.version)

Python 2.7.15 (default, Oct 22 2018, 19:33:46)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]

%%script python3
import sys


print("Python: %s" % sys.version)

Python: 3.11.4 (main, Jun 15 2023, 07:55:38) [Clang 14.0.3 (clang-1403.0.22.14.1)]

%%ruby
puts "Ruby #{RUBY_VERSION}"

Ruby 2.6.10

%%bash
echo "$BASH"

/bin/bash

You can capture stdout and err from these sub-processes in Python variables:

%%bash --out output --err error
echo "stdout"
echo "stderr" >&2

print(error)
print(output)

stderr

stdout

Configure standard script magic

The list of aliases for the script magic is configurable. By default, some common interpreters can be used if necessary. However, you can also specify your own interpreter in ipython_config.py:

c.ScriptMagics.scripts = ["R", "pypy", "myprogram"]
c.ScriptMagics.script_paths = {"myprogram": "/path/to/myprogram"}

Help functions: ?, %magic and %lsmagic

Like normal Python functions, the IPython magic functions have docstrings that can be easily accessed. E.g. to read the documentation of the %timeit magic, just type:

%timeit?

Documentation for other functions can be accessed in a similar manner. To access a general description of the %magic functions available, including some examples, you can type:

%magic

For a quick list of all available magic functions, type:

%lsmagic

Available line magics:
%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

You can also simply define your own magic functions. For more information, see Defining custom magics.