Introduction

This project should help with executing the same tasks on pre-defined scenarios/profiles. In case the tasks provide pbench-like json results it also allows tools to analyze and compare the results with main focus on monitoring performance in time.

The main usecase of this tool is a performance regression CI.

Setup

Run-perf is available from pip so one can install it by executing:

python3 -m pip install runperf

or to install directly the latest version from git:

python3 -m pip install git+https://github.com/distributed-system-analysis/run-perf.git

For development purposes please check-out the clone-and-deploy section in the documentation.

Components

• run-perf => run perf test(s) and report results

• compare-perf => compare 2 or more runperf results together reporting human as well as machine readable output optionally supporting model to smooth the comparisons

• analyze-perf => calculate a model based on one or multiple results

Basic usage

Execute uperf and fio (with custom params) on machine foo.example.org that will be provisioned via beaker to Fedora-32. Execute the tests under Localhost (directly on the foo.example.org machine) and TunedLibvirt (configures host, fetches guest image, configures it and spawns guest VM) profiles and report results in ./result_$date directory:

run-perf -vvv --hosts foo:foo.example.org --provisioner Beaker --distro Fedora-32 --default-password password --profiles Localhost TunedLibvirt -- uperf fio:'{"type":"read", "ramptime":"1", "runtime":"10", "samples":"1", "file-size": "100", "targets": "/fio"}'

Process result* directories, compare the ranges and create a linear model that normalizes the ranges to <-3, +3> range:

analyze-perf -vvv -l model1.json -t 3 -- result*

Compare src and dst results using model1.json linear model and report the comparison in human readable form to the console, in XUNIT format in result.xml file and as a standalone html page in result.html. For some tasks the result* results are also added as reference for better visualization of the changes:

compare-perf -vvv --tolerance 5 --stddev-tolerance 10 -l model1.json --xunit result.xml --html result.html -- src result* dst

---

Table of content

Introduction

• Introduction
  • Setup
  • Components
  • Basic usage
• Run-perf
  • Hosts
  • Profiles
  • Tests
  • Build metadata
• Compare-perf
  • verbose mode
  • html results
• Analyze-perf
  • CSV
  • Linear regression model
• Diff-perf
• Strip-run-perf

Advanced topics

• Results format
  • Common folders
  • Root
  • Profile
  • Tests
  • Individual test
• Using a pre-built container
• Downstream extensions
  • Assets
  • Plugins
• Jenkins integration example
• Contribution guide
  • Reporting Bugs/Enhancments
  • First time contribution
  • Reviewer process

API reference

• runperf package
  • Subpackages
    • runperf.utils package
      • Submodules
      • runperf.utils.cloud_image_providers module
      • runperf.utils.pbench module
      • Module contents
  • Submodules
  • runperf.contrib module
    • get_contrib_path()
    • main()
  • runperf.exceptions module
    • RebootRequest
    • StepFailed
    • TestSkip
  • runperf.html_report module
    • KnownItems
      • KnownItems.add()
      • KnownItems.get_short()
    • anonymize_test_params()
    • generate_report()
  • runperf.machine module
    • BaseMachine
      • BaseMachine.copy_from()
      • BaseMachine.copy_to()
      • BaseMachine.fetch_logs()
      • BaseMachine.get_addr()
      • BaseMachine.get_fullname()
      • BaseMachine.get_host_addr()
      • BaseMachine.get_info()
      • BaseMachine.get_session()
      • BaseMachine.get_session_cont()
      • BaseMachine.get_ssh_cmd()
      • BaseMachine.ssh_copy_id()
    • Controller
      • Controller.apply_profile()
      • Controller.cleanup()
      • Controller.fetch_logs()
      • Controller.for_each_host()
      • Controller.for_each_host_retry()
      • Controller.revert_profile()
      • Controller.run_test()
      • Controller.setup()
      • Controller.write_metadata()
    • Host
      • Host.apply_profile()
      • Host.cleanup()
      • Host.fetch_logs()
      • Host.generate_ssh_key()
      • Host.get_addr()
      • Host.get_fullname()
      • Host.get_host_addr()
      • Host.get_info()
      • Host.get_ssh_cmd()
      • Host.provision()
      • Host.reboot()
      • Host.revert_profile()
      • Host.run_script()
      • Host.setup()
    • LibvirtGuest
      • LibvirtGuest.XML_FILTERS
      • LibvirtGuest.cleanup()
      • LibvirtGuest.get_addr()
      • LibvirtGuest.get_fullname()
      • LibvirtGuest.get_host_addr()
      • LibvirtGuest.get_host_session()
      • LibvirtGuest.get_info()
      • LibvirtGuest.get_ssh_cmd()
      • LibvirtGuest.is_defined()
      • LibvirtGuest.is_running()
      • LibvirtGuest.start()
    • get_distro_info()
  • runperf.profiles module
    • BaseProfile
      • BaseProfile.apply()
      • BaseProfile.fetch_logs()
      • BaseProfile.get_info()
      • BaseProfile.name
      • BaseProfile.revert()
    • DefaultLibvirt
      • DefaultLibvirt.deps
      • DefaultLibvirt.get_info()
      • DefaultLibvirt.img_base
      • DefaultLibvirt.name
    • DefaultLibvirtMulti
      • DefaultLibvirtMulti.name
    • Localhost
      • Localhost.name
    • Overcommit1p5
      • Overcommit1p5.name
    • PersistentProfile
      • PersistentProfile.get_info()
    • TunedLibvirt
      • TunedLibvirt.name
    • get()
  • runperf.provisioners module
    • Beaker
      • Beaker.name
      • Beaker.provision()
  • runperf.result module
    • AveragesModifier
      • AveragesModifier.COEFFICIENT
      • AveragesModifier.add_result()
      • AveragesModifier.check_result()
    • Model
      • Model.check_result()
      • Model.identify()
      • Model.mean_tolerance
      • Model.processing_dst_results
      • Model.stddev_tolerance
    • ModelLinearRegression
      • ModelLinearRegression.TOO_STRICT_COEFFICIENT
      • ModelLinearRegression.check_result()
      • ModelLinearRegression.identify()
      • ModelLinearRegression.rebase()
    • ModelStdev
      • ModelStdev.ERROR_COEFICIENT
      • ModelStdev.TOO_STRICT_COEFFICIENT
      • ModelStdev.identify()
    • Modifier
      • Modifier.add_result()
      • Modifier.check_result()
      • Modifier.weight
    • NOutOfResultsModifier
      • NOutOfResultsModifier.add_result()
      • NOutOfResultsModifier.check_result()
    • RelativeResults
      • RelativeResults.compute_statistics()
      • RelativeResults.evaluate()
      • RelativeResults.expand_grouped_results()
      • RelativeResults.finish()
      • RelativeResults.get_xunit()
      • RelativeResults.per_type_stats()
      • RelativeResults.record()
      • RelativeResults.record_broken()
      • RelativeResults.record_result()
      • RelativeResults.sum_stats()
    • Result
      • Result.add()
      • Result.add_bad()
      • Result.agg_diffs
      • Result.agg_weights
      • Result.big
      • Result.classname
      • Result.details
      • Result.dst
      • Result.error
      • Result.get_merged_name()
      • Result.good
      • Result.is_error()
      • Result.is_stddev()
      • Result.name
      • Result.params
      • Result.primary
      • Result.score
      • Result.small
      • Result.srcs
      • Result.status
      • Result.testname
      • Result.tolerance
    • ResultsContainer
      • ResultsContainer.add_result_by_path()
    • closest_result()
    • get_uncertainty()
    • iter_results()
    • iter_results_errors()
    • iter_results_jsons()
  • runperf.tests module
    • BaseTest
      • BaseTest.cleanup()
      • BaseTest.inject_metadata()
      • BaseTest.min_groups
      • BaseTest.name
      • BaseTest.run()
      • BaseTest.setup()
    • DummyTest
      • DummyTest.name
    • Linpack
      • Linpack.default_args
      • Linpack.name
      • Linpack.test
    • PBenchFio
      • PBenchFio.default_args
      • PBenchFio.name
      • PBenchFio.test
    • PBenchLibblkio
      • PBenchLibblkio.base_path
      • PBenchLibblkio.cleanup()
      • PBenchLibblkio.default_args
      • PBenchLibblkio.name
      • PBenchLibblkio.setup()
    • PBenchNBD
      • PBenchNBD.base_path
      • PBenchNBD.cleanup()
      • PBenchNBD.default_args
      • PBenchNBD.name
      • PBenchNBD.setup()
    • PBenchTest
      • PBenchTest.args
      • PBenchTest.default_args
      • PBenchTest.setup()
      • PBenchTest.test
      • PBenchTest.timeout
      • PBenchTest.watchdog_timeout
    • UPerf
      • UPerf.default_args
      • UPerf.name
      • UPerf.test
    • get()
  • runperf.version module
    • get_version()
  • Module contents
    • AnalyzePerf
    • ComparePerf
    • DictAction
    • DiffPerf
    • StripPerf
      • StripPerf.process_result_json()
      • StripPerf.process_result_metadata()
      • StripPerf.process_sysinfo()
    • create_metadata()
    • get_abs_path()
    • item_with_params()
    • logging_argparse()
    • logging_setup()
    • main()
    • parse_host()
    • profile_test_defs()

Indices and tables

• Index

• Module Index

• Search Page