Usage - Commands

Benchpress provides a set of binaries for running and visualize the suite file suite_file.

bp-run

After you have created a benchmark suite file (see. The Suite File), you can then use bp-run to execute the suite:

usage: run.py [-h] [--nruns NRUNS] [--warmup] [--dirty] [--tag TAG] [--slurm]
              [--partition PARTITION] [--multi-jobs] [--nice NICE]
              suite

Runs a benchmark suite and stores the results in a JSON-file.

positional arguments:
  suite                 Path to the JSON file where the benchmark results will
                        be read and written. If the file exist, the benchmark
                        will resume.

optional arguments:
  -h, --help            show this help message and exit
  --nruns NRUNS         How many times should each command run.
  --warmup              Execute one warm up run, before the measured runs
  --dirty               Do no clean up.
  --tag TAG             Assign a tag to the result.

SLURM Queuing System:
  --slurm               Use the SLURM queuing system.
  --partition PARTITION
                        Submit to a specific SLURM partition.
  --multi-jobs          Submit 'nruns' SLURM jobs instead of one job with
                        'nruns' number of runs.
  --nice NICE           The scheduling priority - range is from -10000
                        (highest priority) to 10000 (lowest priority) where
                        zero is default. Only privileged users can specify a
                        negative priority.

bp-cli

At any point, you can read the result of the finished executions within a benchmark suite file. bp-cli prints the benchmark results to the terminal:

usage: cli.py [-h] [-o FILE] [--parse-regex RegEx] [--py-type {float,int,str}]
              [--labels-to-include RegEx] [--labels-to-exclude RegEx]
              [--label-map RegEx:label,...,RegEx:label] [--csv]
              [--csv-separator sep]
              FILE

Prints the result of a Benchpress JSON-file.

positional arguments:
  FILE                  JSON file containing results

optional arguments:
  -h, --help            show this help message and exit
  -o FILE, --output FILE
                        Write output to FILE. (default: None)
  --parse-regex RegEx   How to parse the result of each run. For each RegEx
                        match, group one is recorded as a result. (default:
                        elapsed-time: ([\d.]+))
  --py-type {float,int,str}
                        The Python data type of the parsed results. (default:
                        float)
  --labels-to-include RegEx
                        All labels that match the RegEx are showed. (default:
                        .*)
  --labels-to-exclude RegEx
                        All labels that match the RegEx are ignored. (default:
                        None)
  --label-map RegEx:label,...,RegEx:label
                        Comma separated list of original-to-new-label names
                        (default: None)
  --csv                 Use the CSV format using 'separator' as the separator.
                        (default: False)
  --csv-separator sep   Use the CSV format using 'sep' as the separator.
                        (default: ,)

bp-cli-series

usage: cli_series.py [-h] [-o FILE] [--parse-regex RegEx]
                     [--py-type {float,int,str}] [--labels-to-include RegEx]
                     [--labels-to-exclude RegEx]
                     [--label-map RegEx:label,...,RegEx:label]
                     [--csv-separator sep]
                     [--meta-key {creation_date_utc,tag}]
                     FILE_LIST [FILE_LIST ...]

Prints the result of a series of Benchpress JSON-files.

positional arguments:
  FILE_LIST             JSON file containing results (accept multiple files)

optional arguments:
  -h, --help            show this help message and exit
  -o FILE, --output FILE
                        Write output to FILE. (default: None)
  --parse-regex RegEx   How to parse the result of each run. For each RegEx
                        match, group one is recorded as a result. (default:
                        elapsed-time: ([\d.]+))
  --py-type {float,int,str}
                        The Python data type of the parsed results. (default:
                        float)
  --labels-to-include RegEx
                        All labels that match the RegEx are showed. (default:
                        .*)
  --labels-to-exclude RegEx
                        All labels that match the RegEx are ignored. (default:
                        None)
  --label-map RegEx:label,...,RegEx:label
                        Comma separated list of original-to-new-label names
                        (default: None)
  --csv-separator sep   Use the CSV format using 'sep' as the separator.
                        (default: ,)
  --meta-key {creation_date_utc,tag}
                        The meta key, which value vary throughout the series
                        (default: creation_date_utc)

bp-chart


bp-chart-series

usage: series_per_cmd.py [-h] [-o FILE] [--parse-regex RegEx]
                         [--py-type {float,int,str}]
                         [--labels-to-include RegEx]
                         [--labels-to-exclude RegEx]
                         [--label-map RegEx:label,...,RegEx:label]
                         [--title TITLE] [--warmups WARMUPS] [--ymax YMAX]
                         [--ymin YMIN] [--ylog] [--ylabel YLABEL]
                         [--plot-size PLOT_SIZE]
                         [--xticklabel-rotation DEGREE] [--no-legend]
                         [--fontsize FONTSIZE] [--pyplot-style PYPLOT_STYLE]
                         [--meta-key {creation_date_utc,tag}] [--mpld3]
                         FILE_LIST [FILE_LIST ...]

Plots the result of a Benchpress JSON-file (one bar per command)

positional arguments:
  FILE_LIST             JSON file containing results (accept multiple files)

optional arguments:
  -h, --help            show this help message and exit
  -o FILE, --output FILE
                        Write output to FILE. (default: None)
  --parse-regex RegEx   How to parse the result of each run. For each RegEx
                        match, group one is recorded as a result. (default:
                        elapsed-time: ([\d.]+))
  --py-type {float,int,str}
                        The Python data type of the parsed results. (default:
                        float)
  --labels-to-include RegEx
                        All labels that match the RegEx are showed. (default:
                        .*)
  --labels-to-exclude RegEx
                        All labels that match the RegEx are ignored. (default:
                        None)
  --label-map RegEx:label,...,RegEx:label
                        Comma separated list of original-to-new-label names
                        (default: None)
  --title TITLE         The title of the chart. (default: None)
  --warmups WARMUPS     Specify the amount of samples from warm-up rounds.
                        (default: 0)
  --ymax YMAX           Max value of the y-axis (default: None)
  --ymin YMIN           Min value of the y-axis (default: None)
  --ylog                Makes the y-axis logarithmic (default: False)
  --ylabel YLABEL       Label on the y-axis (default: Elapsed time in seconds)
  --plot-size PLOT_SIZE
                        Size of the plot compared to total figure (0.1 to 1.0)
                        (default: 0.7)
  --xticklabel-rotation DEGREE
                        The rotation of the labels on the x-axis (default: 90)
  --no-legend           Hide the legend box (default: False)
  --fontsize FONTSIZE   Fontsize (default: 12)
  --pyplot-style PYPLOT_STYLE
                        The matplotlib.pyplot.style to use (default: seaborn-
                        darkgrid)
  --meta-key {creation_date_utc,tag}
                        The meta key, which value vary throughout the series
                        (default: creation_date_utc)
  --mpld3               Generate mpld3 plot <https://mpld3.github.io>
                        (default: False)

bp-raw

usage: raw.py [-h] [-o FILE] [--parse-regex RegEx] [--py-type {float,int,str}]
              [--labels-to-include RegEx] [--labels-to-exclude RegEx]
              [--label-map RegEx:label,...,RegEx:label] [--no-stdout]
              [--no-stderr]
              FILE

Raw dump of the stdout and stderr in a Benchpress JSON-file.

positional arguments:
  FILE                  JSON file containing results

optional arguments:
  -h, --help            show this help message and exit
  -o FILE, --output FILE
                        Write output to FILE. (default: None)
  --parse-regex RegEx   How to parse the result of each run. For each RegEx
                        match, group one is recorded as a result. (default:
                        elapsed-time: ([\d.]+))
  --py-type {float,int,str}
                        The Python data type of the parsed results. (default:
                        float)
  --labels-to-include RegEx
                        All labels that match the RegEx are showed. (default:
                        .*)
  --labels-to-exclude RegEx
                        All labels that match the RegEx are ignored. (default:
                        None)
  --label-map RegEx:label,...,RegEx:label
                        Comma separated list of original-to-new-label names
                        (default: None)
  --no-stdout           Don't dump stdout. (default: False)
  --no-stderr           Don't dump stderr. (default: False)