Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add detailed JSON support to performance summarization script #9172

Draft
wants to merge 2 commits into
base: v2-performance-testing
Choose a base branch
from

Conversation

aemous
Copy link
Contributor

@aemous aemous commented Dec 24, 2024

Description of changes:

  • Added a new detailed_json output format to the performance summarization script. It outputs measurements from all iterations of the command instead of just the average / maxima.
  • Added extra metadata that gets output with each metric which is useful for downstream processing of the metrics.
  • Changed Summarizer class so that it returns the raw output data in summarize_as_json and summarize_as_detailed_json functions. Useful for external classes to utilize this data for processing.

Description of tests:

  • Ran the following commands to benchmark a file upload with aws s3 cp and summarize the results in the new detailed JSON format:
% ./benchmark "aws s3 cp ~/file.dmg s3://test-bucket-aem278/file.dmg" --data-interval 0.1 --output-file perf1.csv
% ./benchmark "aws s3 cp ~/file.dmg s3://test-bucket-aem278/file.dmg" --data-interval 0.1 --output-file perf2.csv
% ./benchmark "aws s3 cp ~/file.dmg s3://test-bucket-aem278/file.dmg" --data-interval 0.1 --output-file perf3.csv
% python3 ./summarize.py perf1.csv perf2.csv perf3.csv --output-format detailed_json --benchmark-name s3.cp-upload

Output:

[
  {
    "name": "s3.cp-upload.total_time",
    "description": "The total execution time of the command.",
    "unit": "Milliseconds",
    "date": 1735074973.303767,
    "measurements": [
      354898.06270599365,
      355706.1140537262,
      348326.67684555054
    ]
  },
  {
    "name": "s3.cp-upload.max_memory",
    "description": "The maximum memory utilization of the CLI throughout execution of the command.",
    "unit": "Megabytes",
    "date": 1735074973.303767,
    "measurements": [
      123.338752,
      134.463488,
      130.023424
    ]
  },
  {
    "name": "s3.cp-upload.average_memory",
    "description": "The average memory utilization of the CLI throughout execution of the command.",
    "unit": "Megabytes",
    "date": 1735074973.303767,
    "measurements": [
      122.78925544327932,
      117.97557115870967,
      128.00244748641975
    ]
  },
  {
    "name": "s3.cp-upload.max_cpu_utilization",
    "description": "The maximum CPU utilization of the CLI throughout execution of the command.",
    "unit": "Percentage",
    "date": 1735074973.303767,
    "measurements": [
      3.1,
      3.0,
      4.2
    ]
  },
  {
    "name": "s3.cp-upload.average_cpu_utilization",
    "description": "The average CPU utilization of the CLI throughout execution of the command.",
    "unit": "Percentage",
    "date": 1735074973.303767,
    "measurements": [
      0.008484270734032429,
      0.026161290322580504,
      0.07978835978835712
    ]
  }
]

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant