scalene#

scalene creates profiles for CPU and memory very quickly. The overhead is usually very low at 10–20%.

See also

Installation#

Linux, MacOS and WSL:

$ pipenv install scalene

Use#

  1. An example programme for profiling

[1]:

import numpy as np


def profile_me():
    for i in range(6):
        x = np.array(range(10**7))
        y = np.array(np.random.uniform(0, 100, size=(10**8)))

  1. Load scalene

[2]:

%load_ext scalene

Scalene extension successfully loaded. Note: Scalene currently only
supports CPU+GPU profiling inside Jupyter notebooks. For full Scalene
profiling, use the command line version.

NOTE: in Jupyter notebook on MacOS, Scalene cannot profile child
processes. Do not run to try Scalene with multiprocessing in Jupyter
Notebook.

  1. Profile with only one line of code

[ ]:

%scrun profile_me()
import numpy as np


def profile_me():
    for i in range(6):
        x = np.array(range(10**7))
        y = np.array(np.random.uniform(0, 100, size=(10**8)))

Create a reduced profile (only rows with non-zero counts)

[ ]:

%scrun --reduced-profile profile_me()
import numpy as np


def profile_me():
    for i in range(6):
        x = np.array(range(10**7))
        y = np.array(np.random.uniform(0, 100, size=(10**8)))

For a complete list of options, contact:

[5]:

%scrun --help

usage: scalene [-h] [--version] [--column-width COLUMN_WIDTH]
               [--outfile OUTFILE] [--html] [--json] [--cli] [--stacks]
               [--web] [--viewer] [--reduced-profile]
               [--profile-interval PROFILE_INTERVAL] [--cpu] [--cpu-only]
               [--gpu] [--memory] [--profile-all]
               [--profile-only PROFILE_ONLY]
               [--profile-exclude PROFILE_EXCLUDE] [--use-virtual-time]
               [--cpu-percent-threshold CPU_PERCENT_THRESHOLD]
               [--cpu-sampling-rate CPU_SAMPLING_RATE]
               [--allocation-sampling-window ALLOCATION_SAMPLING_WINDOW]
               [--malloc-threshold MALLOC_THRESHOLD]
               [--program-path PROGRAM_PATH] [--memory-leak-detector]
               [--on | --off]

Scalene: a high-precision CPU and memory profiler, version 1.5.23 (2023.07.26)
https://github.com/plasma-umass/scalene

command-line:
  % scalene [options] your_program.py [--- --your_program_args] 
or
  % python3 -m scalene [options] your_program.py [--- --your_program_args] 

in Jupyter, line mode:
  %scrun [options] statement

in Jupyter, cell mode:
  %%scalene [options]
   your code here


options:
  -h, --help            show this help message and exit
  --version             prints the version number for this release of Scalene and exits
  --column-width COLUMN_WIDTH
                        Column width for profile output (default: 132)
  --outfile OUTFILE     file to hold profiler output (default: stdout)
  --html                output as HTML (default: web)
  --json                output as JSON (default: web)
  --cli                 forces use of the command-line
  --stacks              collect stack traces
  --web                 opens a web tab to view the profile (saved as 'profile.html')
  --viewer              only opens the web UI (https://plasma-umass.org/scalene-gui/)
  --reduced-profile     generate a reduced profile, with non-zero lines only (default: False)
  --profile-interval PROFILE_INTERVAL
                        output profiles every so many seconds (default: inf)
  --cpu                 profile CPU time (default:  True )
  --cpu-only            profile CPU time (deprecated: use --cpu )
  --gpu                 profile GPU time and memory (default: False )
  --memory              profile memory (default: True )
  --profile-all         profile all executed code, not just the target program (default: only the target program)
  --profile-only PROFILE_ONLY
                        profile only code in filenames that contain the given strings, separated by commas
(default: no restrictions)
  --profile-exclude PROFILE_EXCLUDE
                        do not profile code in filenames that contain the given strings, separated by commas
(default: no restrictions)
  --use-virtual-time    measure only CPU time, not time spent in I/O or blocking (default: False)
  --cpu-percent-threshold CPU_PERCENT_THRESHOLD
                        only report profiles with at least this percent of CPU time (default: 1%)
  --cpu-sampling-rate CPU_SAMPLING_RATE
                        CPU sampling rate (default: every 0.01s)
  --allocation-sampling-window ALLOCATION_SAMPLING_WINDOW
                        Allocation sampling window size, in bytes (default: 10485767 bytes)
  --malloc-threshold MALLOC_THRESHOLD
                        only report profiles with at least this many allocations (default: 100)
  --program-path PROGRAM_PATH
                        The directory containing the code to profile (default: the path to the profiled program)
  --memory-leak-detector
                        EXPERIMENTAL: report likely memory leaks (default: True)
  --on                  start with profiling on (default)
  --off                 start with profiling off

When running Scalene in the background, you can suspend/resume profiling
for the process ID that Scalene reports. For example:

   % python3 -m scalene  yourprogram.py &
 Scalene now profiling process 12345
   to suspend profiling: python3 -m scalene.profile --off --pid 12345
   to resume profiling:  python3 -m scalene.profile --on  --pid 12345

Profile with more than one line of code in a cell

[ ]:

%%scalene --reduced-profile
x = 0
for i in range(1000):
    for j in range(1000):
        x += 1