{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IPython magic\n",
    "\n",
    "IPython not only enables Python to be used interactively, but also extends the Python syntax with so-called *magic commands*, which are provided with the prefix `%`. They are designed to quickly and easily solve common data analysis problems. A distinction is made between two different types of *magic commands*:\n",
    "\n",
    "* *line magics*, denoted by a single `%` prefix, that run on a single input line\n",
    "* *cell magics* which are preceded by a double symbol `%%` and which are executed within a notebook cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute external code: `%run`\n",
    "\n",
    "If you start developing larger code, you will likely be working in both IPython for interactive exploration and a text editor to save code that you want to reuse. With the `%run` magic you can execute this code directly in your IPython session.\n",
    "\n",
    "Imagine you created a `myscript.py` file with the following content:\n",
    "\n",
    "``` Python\n",
    "def square(x):\n",
    "    return x**2\n",
    "\n",
    "\n",
    "for N in range(1, 4):\n",
    "    print(N, \"squared is\", square(N))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 squared is 1\n",
      "2 squared is 4\n",
      "3 squared is 9\n"
     ]
    }
   ],
   "source": [
    "%run myscript.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that after running this script, all of the functions defined in it will be available for use in your IPython session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "square(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several ways you can improve the way your code runs. As usual, you can display the documentation in IPython with `%run?`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run timing code: `%timeit`\n",
    "\n",
    "Another example of a Magic function is `%timeit`, which automatically determines the execution time of the following one-line Python statement. So we can e.g. output the performance of a list comprehension with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27.9 μs ± 286 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit L = [n ** 2 for n in range(1000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The advantage of `%timeit` is that short commands automatically run multiple runs to get more robust results. For multi-line instructions, adding a second `%` character creates cell magic that can process multiple input lines. For example, here is the equivalent construction using a `for` loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31.1 μs ± 164 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "L = []\n",
    "for n in range(1000):\n",
    "    L.append(n ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can immediately see that the list comprehension is about 10% faster than its equivalent with a `for` loop. We then describe performance measurements and optimisations in more detail in [Profiling](../../performance/ipython-profiler.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute code from other interpreters\n",
    "\n",
    "IPython has a `%%script` script magic with which you can execute a cell in a subprocess of an interpreter on your system, e.g. `bash`, `ruby`, `perl`, `zsh`, `R` etc. This can also be its own script that expects input in `stdin`. To do this, simply pass a path or a shell command to the program that is specified in the `%%script` line. The rest of the cell is executed by this script, capturing `stdout` or `err` from the subprocess and displaying it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ruby 3.3.5\n"
     ]
    }
   ],
   "source": [
    "%%ruby\n",
    "puts \"Ruby #{RUBY_VERSION}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/homebrew/bin/bash\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo \"$BASH\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can capture `stdout` and `err` from these sub-processes in Python variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash --out output --err error\n",
    "echo \"stdout\"\n",
    "echo \"stderr\" >&2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stderr\n",
      "\n",
      "stdout\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(error)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure standard `script` magic\n",
    "\n",
    "The list of aliases for the `script` magic is configurable. By default, some common interpreters can be used if necessary. However, you can also specify your own interpreter in `ipython_config.py`:\n",
    "``` Python\n",
    "c.ScriptMagics.scripts = [\"R\", \"pypy\", \"myprogram\"]\n",
    "c.ScriptMagics.script_paths = {\"myprogram\": \"/path/to/myprogram\"}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Help functions: `?`, `%magic` and `%lsmagic`\n",
    "\n",
    "Like normal Python functions, the IPython magic functions have docstrings that can be easily accessed. E.g. to read the documentation of the `%timeit` magic, just type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Time execution of a Python statement or expression\n",
       "\n",
       "**Usage, in line mode:**\n",
       "\n",
       "  %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement\n",
       "\n",
       "**or in cell mode:**\n",
       "\n",
       "  %%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code\n",
       "\n",
       "  code\n",
       "\n",
       "  code...\n",
       "\n",
       "Time execution of a Python statement or expression using the timeit\n",
       "module.  This function can be used both as a line and cell magic:\n",
       "\n",
       "- In line mode you can time a single-line statement (though multiple\n",
       "  ones can be chained with using semicolons).\n",
       "\n",
       "- In cell mode, the statement in the first line is used as setup code\n",
       "  (executed but not timed) and the body of the cell is timed.  The cell\n",
       "  body has access to any variables created in the setup code.\n",
       "\n",
       "…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%timeit?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Documentation for other functions can be accessed in a similar manner. To access a general description of the `%magic` functions available, including some examples, you can type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "IPython's 'magic' functions\n",
       "===========================\n",
       "\n",
       "The magic function system provides a series of functions which allow you to\n",
       "control the behavior of IPython itself, plus a lot of system-type\n",
       "features. There are two kinds of magics, line-oriented and cell-oriented.\n",
       "\n",
       "Line magics are prefixed with the % character and work much like OS\n",
       "command-line calls: they get as an argument the rest of the line, where\n",
       "arguments are passed without parentheses or quotes.  For example, this will\n",
       "time the given statement::\n",
       "\n",
       "        %timeit range(1000)\n",
       "\n",
       "Cell magics are prefixed with a double %%, and they are functions that get as\n",
       "an argument not only the rest of the line, but also the lines below it in a\n",
       "separate argument.  These magics are called with two arguments: the rest of the\n",
       "call line and the body of the cell, consisting of the lines below the first.\n",
       "For example::\n",
       "\n",
       "        %%timeit x = numpy.random.randn((100, 100))\n",
       "        numpy.linalg.svd(x)\n",
       "\n",
       "will time the execution of the numpy svd routine, running the assignment of x\n",
       "as part of the setup phase, which is not timed.\n",
       "\n",
       "…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%magic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a quick list of all available `magic` functions, type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "cell": {
        "!": "OSMagics",
        "HTML": "Other",
        "SVG": "Other",
        "bash": "Other",
        "capture": "ExecutionMagics",
        "code_wrap": "ExecutionMagics",
        "debug": "ExecutionMagics",
        "file": "Other",
        "html": "DisplayMagics",
        "javascript": "DisplayMagics",
        "js": "DisplayMagics",
        "latex": "DisplayMagics",
        "markdown": "DisplayMagics",
        "perl": "Other",
        "prun": "ExecutionMagics",
        "pypy": "Other",
        "python": "Other",
        "python2": "Other",
        "python3": "Other",
        "ruby": "Other",
        "script": "ScriptMagics",
        "sh": "Other",
        "svg": "DisplayMagics",
        "sx": "OSMagics",
        "system": "OSMagics",
        "time": "ExecutionMagics",
        "timeit": "ExecutionMagics",
        "writefile": "OSMagics"
       },
       "line": {
        "alias": "OSMagics",
        "alias_magic": "BasicMagics",
        "autoawait": "AsyncMagics",
        "autocall": "AutoMagics",
        "automagic": "AutoMagics",
        "autosave": "KernelMagics",
        "bookmark": "OSMagics",
        "cat": "Other",
        "cd": "OSMagics",
        "clear": "KernelMagics",
        "code_wrap": "ExecutionMagics",
        "colors": "BasicMagics",
        "conda": "PackagingMagics",
        "config": "ConfigMagics",
        "connect_info": "KernelMagics",
        "cp": "Other",
        "debug": "ExecutionMagics",
        "dhist": "OSMagics",
        "dirs": "OSMagics",
        "doctest_mode": "BasicMagics",
        "ed": "Other",
        "edit": "KernelMagics",
        "env": "OSMagics",
        "gui": "BasicMagics",
        "hist": "Other",
        "history": "HistoryMagics",
        "killbgscripts": "ScriptMagics",
        "ldir": "Other",
        "less": "KernelMagics",
        "lf": "Other",
        "lk": "Other",
        "ll": "Other",
        "load": "CodeMagics",
        "load_ext": "ExtensionMagics",
        "loadpy": "CodeMagics",
        "logoff": "LoggingMagics",
        "logon": "LoggingMagics",
        "logstart": "LoggingMagics",
        "logstate": "LoggingMagics",
        "logstop": "LoggingMagics",
        "ls": "Other",
        "lsmagic": "BasicMagics",
        "lx": "Other",
        "macro": "ExecutionMagics",
        "magic": "BasicMagics",
        "mamba": "PackagingMagics",
        "man": "KernelMagics",
        "matplotlib": "PylabMagics",
        "micromamba": "PackagingMagics",
        "mkdir": "Other",
        "more": "KernelMagics",
        "mv": "Other",
        "notebook": "BasicMagics",
        "page": "BasicMagics",
        "pastebin": "CodeMagics",
        "pdb": "ExecutionMagics",
        "pdef": "NamespaceMagics",
        "pdoc": "NamespaceMagics",
        "pfile": "NamespaceMagics",
        "pinfo": "NamespaceMagics",
        "pinfo2": "NamespaceMagics",
        "pip": "PackagingMagics",
        "popd": "OSMagics",
        "pprint": "BasicMagics",
        "precision": "BasicMagics",
        "prun": "ExecutionMagics",
        "psearch": "NamespaceMagics",
        "psource": "NamespaceMagics",
        "pushd": "OSMagics",
        "pwd": "OSMagics",
        "pycat": "OSMagics",
        "pylab": "PylabMagics",
        "qtconsole": "KernelMagics",
        "quickref": "BasicMagics",
        "recall": "HistoryMagics",
        "rehashx": "OSMagics",
        "reload_ext": "ExtensionMagics",
        "rep": "Other",
        "rerun": "HistoryMagics",
        "reset": "NamespaceMagics",
        "reset_selective": "NamespaceMagics",
        "rm": "Other",
        "rmdir": "Other",
        "run": "ExecutionMagics",
        "save": "CodeMagics",
        "sc": "OSMagics",
        "set_env": "OSMagics",
        "store": "StoreMagics",
        "sx": "OSMagics",
        "system": "OSMagics",
        "tb": "ExecutionMagics",
        "time": "ExecutionMagics",
        "timeit": "ExecutionMagics",
        "unalias": "OSMagics",
        "unload_ext": "ExtensionMagics",
        "uv": "PackagingMagics",
        "who": "NamespaceMagics",
        "who_ls": "NamespaceMagics",
        "whos": "NamespaceMagics",
        "xdel": "NamespaceMagics",
        "xmode": "BasicMagics"
       }
      },
      "text/plain": [
       "Available line magics:\n",
       "%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %code_wrap  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %mamba  %man  %matplotlib  %micromamba  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %uv  %who  %who_ls  %whos  %xdel  %xmode\n",
       "\n",
       "Available cell magics:\n",
       "%%!  %%HTML  %%SVG  %%bash  %%capture  %%code_wrap  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile\n",
       "\n",
       "Automagic is ON, % prefix IS NOT needed for line magics."
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%lsmagic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also simply define your own `magic` functions. For more information, see [Defining custom magics](https://ipython.readthedocs.io/en/stable/config/custommagics.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.13 Kernel",
   "language": "python",
   "name": "python313"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}