Usage - Commands¶
Benchpress provides a set of binaries for running and visualize the suite file The Suite File.
bp-run¶
After you have created a benchmark suite file (see. The Suite File), you can then use bp-run to execute the suite:
usage: run.py [-h] [--nruns NRUNS] [--warmup] [--dirty] [--tag TAG] [--slurm]
[--partition PARTITION] [--multi-jobs] [--nice NICE]
suite
Runs a benchmark suite and stores the results in a JSON-file.
positional arguments:
suite Path to the JSON file where the benchmark results will
be read and written. If the file exist, the benchmark
will resume.
optional arguments:
-h, --help show this help message and exit
--nruns NRUNS How many times should each command run.
--warmup Execute one warm up run, before the measured runs
--dirty Do no clean up.
--tag TAG Assign a tag to the result.
SLURM Queuing System:
--slurm Use the SLURM queuing system.
--partition PARTITION
Submit to a specific SLURM partition.
--multi-jobs Submit 'nruns' SLURM jobs instead of one job with
'nruns' number of runs.
--nice NICE The scheduling priority - range is from -10000
(highest priority) to 10000 (lowest priority) where
zero is default. Only privileged users can specify a
negative priority.
bp-cli¶
At any point, you can read the result of the finished executions within a benchmark suite file.
bp-cli prints the benchmark results to the terminal:
usage: cli.py [-h] [-o FILE] [--parse-regex RegEx] [--py-type {float,int,str}]
[--labels-to-include RegEx] [--labels-to-exclude RegEx]
[--label-map RegEx:label,...,RegEx:label] [--csv]
[--csv-separator sep]
FILE
Prints the result of a Benchpress JSON-file.
positional arguments:
FILE JSON file containing results
optional arguments:
-h, --help show this help message and exit
-o FILE, --output FILE
Write output to FILE. (default: None)
--parse-regex RegEx How to parse the result of each run. For each RegEx
match, group one is recorded as a result. (default:
elapsed-time: ([\d.]+))
--py-type {float,int,str}
The Python data type of the parsed results. (default:
float)
--labels-to-include RegEx
All labels that match the RegEx are showed. (default:
.*)
--labels-to-exclude RegEx
All labels that match the RegEx are ignored. (default:
None)
--label-map RegEx:label,...,RegEx:label
Comma separated list of original-to-new-label names
(default: None)
--csv Use the CSV format using 'separator' as the separator.
(default: False)
--csv-separator sep Use the CSV format using 'sep' as the separator.
(default: ,)
bp-cli-series¶
usage: cli_series.py [-h] [-o FILE] [--parse-regex RegEx]
[--py-type {float,int,str}] [--labels-to-include RegEx]
[--labels-to-exclude RegEx]
[--label-map RegEx:label,...,RegEx:label]
[--csv-separator sep]
[--meta-key {creation_date_utc,tag}]
FILE_LIST [FILE_LIST ...]
Prints the result of a series of Benchpress JSON-files.
positional arguments:
FILE_LIST JSON file containing results (accept multiple files)
optional arguments:
-h, --help show this help message and exit
-o FILE, --output FILE
Write output to FILE. (default: None)
--parse-regex RegEx How to parse the result of each run. For each RegEx
match, group one is recorded as a result. (default:
elapsed-time: ([\d.]+))
--py-type {float,int,str}
The Python data type of the parsed results. (default:
float)
--labels-to-include RegEx
All labels that match the RegEx are showed. (default:
.*)
--labels-to-exclude RegEx
All labels that match the RegEx are ignored. (default:
None)
--label-map RegEx:label,...,RegEx:label
Comma separated list of original-to-new-label names
(default: None)
--csv-separator sep Use the CSV format using 'sep' as the separator.
(default: ,)
--meta-key {creation_date_utc,tag}
The meta key, which value vary throughout the series
(default: creation_date_utc)
bp-chart¶
usage: bar_per_cmd.py [-h] [-o FILE] [--parse-regex RegEx]
[--py-type {float,int,str}] [--labels-to-include RegEx]
[--labels-to-exclude RegEx]
[--label-map RegEx:label,...,RegEx:label]
[--title TITLE] [--warmups WARMUPS] [--ymax YMAX]
[--ymin YMIN] [--ylog] [--ylabel YLABEL]
[--xticklabel-rotation DEGREE] [--no-legend]
[--fontsize FONTSIZE]
[--pyplot-style {seaborn-darkgrid,Solarize_Light2,seaborn-notebook,classic,seaborn-ticks,grayscale,bmh,seaborn-talk,dark_background,ggplot,fivethirtyeight,_classic_test,seaborn-colorblind,seaborn-deep,seaborn-whitegrid,seaborn,seaborn-poster,seaborn-bright,seaborn-muted,seaborn-paper,seaborn-white,fast,seaborn-pastel,seaborn-dark,tableau-colorblind10,seaborn-dark-palette}]
[--mpld3]
FILE
Plots the result of a Benchpress JSON-file (one bar per command)
positional arguments:
FILE JSON file containing results
optional arguments:
-h, --help show this help message and exit
-o FILE, --output FILE
Write output to FILE. (default: None)
--parse-regex RegEx How to parse the result of each run. For each RegEx
match, group one is recorded as a result. (default:
elapsed-time: ([\d.]+))
--py-type {float,int,str}
The Python data type of the parsed results. (default:
float)
--labels-to-include RegEx
All labels that match the RegEx are showed. (default:
.*)
--labels-to-exclude RegEx
All labels that match the RegEx are ignored. (default:
None)
--label-map RegEx:label,...,RegEx:label
Comma separated list of original-to-new-label names
(default: None)
--title TITLE The title of the chart. (default: None)
--warmups WARMUPS Specify the amount of samples from warm-up rounds.
(default: 0)
--ymax YMAX Max value of the y-axis (default: None)
--ymin YMIN Min value of the y-axis (default: None)
--ylog Makes the y-axis logarithmic (default: False)
--ylabel YLABEL Label on the y-axis (default: Elapsed time in seconds)
--xticklabel-rotation DEGREE
The rotation of the labels on the x-axis (default: 90)
--no-legend Hide the legend box (default: False)
--fontsize FONTSIZE Fontsize (default: 12)
--pyplot-style {seaborn-darkgrid,Solarize_Light2,seaborn-notebook,classic,seaborn-ticks,grayscale,bmh,seaborn-talk,dark_background,ggplot,fivethirtyeight,_classic_test,seaborn-colorblind,seaborn-deep,seaborn-whitegrid,seaborn,seaborn-poster,seaborn-bright,seaborn-muted,seaborn-paper,seaborn-white,fast,seaborn-pastel,seaborn-dark,tableau-colorblind10,seaborn-dark-palette}
The matplotlib.pyplot.style to use (default: seaborn-
darkgrid)
--mpld3 Generate mpld3 plots <https://mpld3.github.io>
(default: False)
bp-chart-series¶
usage: series_per_cmd.py [-h] [-o FILE] [--parse-regex RegEx]
[--py-type {float,int,str}]
[--labels-to-include RegEx]
[--labels-to-exclude RegEx]
[--label-map RegEx:label,...,RegEx:label]
[--title TITLE] [--warmups WARMUPS] [--ymax YMAX]
[--ymin YMIN] [--ylog] [--ylabel YLABEL]
[--plot-size PLOT_SIZE]
[--xticklabel-rotation DEGREE] [--no-legend]
[--fontsize FONTSIZE] [--pyplot-style PYPLOT_STYLE]
[--meta-key {creation_date_utc,tag}] [--mpld3]
FILE_LIST [FILE_LIST ...]
Plots the result of a Benchpress JSON-file (one bar per command)
positional arguments:
FILE_LIST JSON file containing results (accept multiple files)
optional arguments:
-h, --help show this help message and exit
-o FILE, --output FILE
Write output to FILE. (default: None)
--parse-regex RegEx How to parse the result of each run. For each RegEx
match, group one is recorded as a result. (default:
elapsed-time: ([\d.]+))
--py-type {float,int,str}
The Python data type of the parsed results. (default:
float)
--labels-to-include RegEx
All labels that match the RegEx are showed. (default:
.*)
--labels-to-exclude RegEx
All labels that match the RegEx are ignored. (default:
None)
--label-map RegEx:label,...,RegEx:label
Comma separated list of original-to-new-label names
(default: None)
--title TITLE The title of the chart. (default: None)
--warmups WARMUPS Specify the amount of samples from warm-up rounds.
(default: 0)
--ymax YMAX Max value of the y-axis (default: None)
--ymin YMIN Min value of the y-axis (default: None)
--ylog Makes the y-axis logarithmic (default: False)
--ylabel YLABEL Label on the y-axis (default: Elapsed time in seconds)
--plot-size PLOT_SIZE
Size of the plot compared to total figure (0.1 to 1.0)
(default: 0.7)
--xticklabel-rotation DEGREE
The rotation of the labels on the x-axis (default: 90)
--no-legend Hide the legend box (default: False)
--fontsize FONTSIZE Fontsize (default: 12)
--pyplot-style PYPLOT_STYLE
The matplotlib.pyplot.style to use (default: seaborn-
darkgrid)
--meta-key {creation_date_utc,tag}
The meta key, which value vary throughout the series
(default: creation_date_utc)
--mpld3 Generate mpld3 plot <https://mpld3.github.io>
(default: False)
bp-raw¶
usage: raw.py [-h] [-o FILE] [--parse-regex RegEx] [--py-type {float,int,str}]
[--labels-to-include RegEx] [--labels-to-exclude RegEx]
[--label-map RegEx:label,...,RegEx:label] [--no-stdout]
[--no-stderr]
FILE
Raw dump of the stdout and stderr in a Benchpress JSON-file.
positional arguments:
FILE JSON file containing results
optional arguments:
-h, --help show this help message and exit
-o FILE, --output FILE
Write output to FILE. (default: None)
--parse-regex RegEx How to parse the result of each run. For each RegEx
match, group one is recorded as a result. (default:
elapsed-time: ([\d.]+))
--py-type {float,int,str}
The Python data type of the parsed results. (default:
float)
--labels-to-include RegEx
All labels that match the RegEx are showed. (default:
.*)
--labels-to-exclude RegEx
All labels that match the RegEx are ignored. (default:
None)
--label-map RegEx:label,...,RegEx:label
Comma separated list of original-to-new-label names
(default: None)
--no-stdout Don't dump stdout. (default: False)
--no-stderr Don't dump stderr. (default: False)