IPython magic

IPython not only enables Python to be used interactively, but also extends the Python syntax with so-called magic commands, which are provided with the prefix %. They are designed to quickly and easily solve common data analysis problems. A distinction is made between two different types of magic commands:

• line magics, denoted by a single % prefix, that run on a single input line

• cell magics which are preceded by a double symbol %% and which are executed within a notebook cell.

Execute external code: %run

If you start developing larger code, you will likely be working in both IPython for interactive exploration and a text editor to save code that you want to reuse. With the %run magic you can execute this code directly in your IPython session.

Imagine you created a myscript.py file with the following content:

def square(x):
    return x**2


for N in range(1, 4):
    print(N, "squared is", square(N))

%run myscript.py

1 squared is 1
2 squared is 4
3 squared is 9

Note that after running this script, all of the functions defined in it will be available for use in your IPython session:

square(4)

16

There are several ways you can improve the way your code runs. As usual, you can display the documentation in IPython with %run?.

Run timing code: %timeit

Another example of a Magic function is %timeit, which automatically determines the execution time of the following one-line Python statement. So we can e.g. output the performance of a list comprehension with:

%timeit L = [n ** 2 for n in range(1000)]

27.9 μs ± 286 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

The advantage of %timeit is that short commands automatically run multiple runs to get more robust results. For multi-line instructions, adding a second % character creates cell magic that can process multiple input lines. For example, here is the equivalent construction using a for loop:

%%timeit
L = []
for n in range(1000):
    L.append(n ** 2)

31.1 μs ± 164 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

We can immediately see that the list comprehension is about 10% faster than its equivalent with a for loop. We then describe performance measurements and optimisations in more detail in Profiling.

Execute code from other interpreters

IPython has a %%script script magic with which you can execute a cell in a subprocess of an interpreter on your system, e.g. bash, ruby, perl, zsh, R etc. This can also be its own script that expects input in stdin. To do this, simply pass a path or a shell command to the program that is specified in the %%script line. The rest of the cell is executed by this script, capturing stdout or err from the subprocess and displaying it.

%%ruby
puts "Ruby #{RUBY_VERSION}"

Ruby 3.3.5

%%bash
echo "$BASH"

/opt/homebrew/bin/bash

You can capture stdout and err from these sub-processes in Python variables:

%%bash --out output --err error
echo "stdout"
echo "stderr" >&2

print(error)
print(output)

stderr

stdout

Configure standard script magic

The list of aliases for the script magic is configurable. By default, some common interpreters can be used if necessary. However, you can also specify your own interpreter in ipython_config.py:

c.ScriptMagics.scripts = ["R", "pypy", "myprogram"]
c.ScriptMagics.script_paths = {"myprogram": "/path/to/myprogram"}

Help functions: ?, %magic and %lsmagic

Like normal Python functions, the IPython magic functions have docstrings that can be easily accessed. E.g. to read the documentation of the %timeit magic, just type:

%timeit?

Docstring:
Time execution of a Python statement or expression

**Usage, in line mode:**

  %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement

**or in cell mode:**

  %%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code

  code

  code...

Time execution of a Python statement or expression using the timeit
module.  This function can be used both as a line and cell magic:

- In line mode you can time a single-line statement (though multiple
  ones can be chained with using semicolons).

- In cell mode, the statement in the first line is used as setup code
  (executed but not timed) and the body of the cell is timed.  The cell
  body has access to any variables created in the setup code.

…

Documentation for other functions can be accessed in a similar manner. To access a general description of the %magic functions available, including some examples, you can type:

%magic

IPython's 'magic' functions
===========================

The magic function system provides a series of functions which allow you to
control the behavior of IPython itself, plus a lot of system-type
features. There are two kinds of magics, line-oriented and cell-oriented.

Line magics are prefixed with the % character and work much like OS
command-line calls: they get as an argument the rest of the line, where
arguments are passed without parentheses or quotes.  For example, this will
time the given statement::

        %timeit range(1000)

Cell magics are prefixed with a double %%, and they are functions that get as
an argument not only the rest of the line, but also the lines below it in a
separate argument.  These magics are called with two arguments: the rest of the
call line and the body of the cell, consisting of the lines below the first.
For example::

        %%timeit x = numpy.random.randn((100, 100))
        numpy.linalg.svd(x)

will time the execution of the numpy svd routine, running the assignment of x
as part of the setup phase, which is not timed.

…

For a quick list of all available magic functions, type:

%lsmagic

Available line magics:
%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %code_wrap  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %mamba  %man  %matplotlib  %micromamba  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %uv  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%code_wrap  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

You can also simply define your own magic functions. For more information, see Defining custom magics.