Using mutlity part upload cpu is too high

[jiashiwen]

My code is

 let mut part_number = 0;
    let mut upload_parts: Vec<CompletedPart> = Vec::new();
    let file_len = file.metadata().unwrap().len();
    let mut readed = 0;


    let multipart_upload_res = client
        .create_multipart_upload()
        .bucket(bucket)
        .key(key)
        .send()
        .await?;

    let upload_id = multipart_upload_res.upload_id().unwrap();
    println!("upload_id is {}", upload_id);

    loop {
        let mut buf = vec![0; chunk_size];
        let read_count = file.read(&mut buf).unwrap();
        readed += read_count;
        println!("readed/total:{}/{}", readed, file_len);
        part_number += 1;

        if read_count == 0 {
            break;
        }

        let body = &buf[..read_count];
        let stream = ByteStream::from(body.to_vec());
        let upload_part_res = client
            .upload_part()
            .key(key)
            .bucket(bucket)
            .upload_id(upload_id)
            .body(stream)
            .part_number(part_number)
            .send()
            .await?;

        let completed_part = CompletedPart::builder()
            .e_tag(upload_part_res.e_tag.unwrap_or_default())
            .part_number(part_number)
            .build();

        upload_parts.push(completed_part);

        if read_count != chunk_size {
            break;
        }
    }

    let completed_multipart_upload: CompletedMultipartUpload = CompletedMultipartUpload::builder()
        .set_parts(Some(upload_parts))
        .build();

    let _complete_multipart_upload_res = client
        .complete_multipart_upload()
        .bucket(bucket)
        .key(key)
        .multipart_upload(completed_multipart_upload)
        .upload_id(upload_id)
        .send()
        .await?;
    Ok(())

From flamegraph

aws_sigv4::http_request::canonical_request::CanonicalRequest::payload_hash
aws_sigv4::sign::v4::sha256_hex_string
::update
<digest::core_api::wrapper::CoreWrapper as digest::Update>::update
block_buffer::BlockBuffer<BlockSize,Kind>::digest_blocks
<digest::core_api::wrapper::CoreWrapper as digest::Update>::update::{{closure}}
<digest::core_api::ct_variable::CtVariableCoreWrapper<T,OutSize,O> as digest::core_api::UpdateCore>::update_blocks
<sha2::core_api::Sha256VarCore as digest::core_api::UpdateCore>::update_blocks

use about 70% cpu,may be cpu used to compress request.
From aws https://docs.aws.amazon.com/sdkref/latest/guide/feature-compression.html,some parameter can disable compress,how to do with rust sdk?

[jiashiwen]

[rcoh]

Interesting. It seems like the CPU is used computing the SHA256 checksum (the compression is a little misleading here, that's just a name of function inside of sha256).

Can you say more about the platform you're running on? This flamegraph is definitely from a release build right? We've done a lot of benchmarking of multipart upload (including in parallel) and we didn't see this behavior.

[jmklix]

Marking this as answered for now because of no response. Please open a new discussion if you are still having problems with cpu being too high.

[shikhar]

We have also observed this behaviour running on Graviton, and it turned out to be because of requiring the asm crate feature on the sha2 crate to enable hardware acceleration of SHA256. By adding an explicit dependency on sha2 with that crate feature, we saw CPU usage from sha256 cut by half.

The asm crate feature is gone in 0.11.x (unreleased) - in the meantime, perhaps it should be included in the SDK's dependency?

https://github.com/RustCrypto/hashes/blob/master/sha2/CHANGELOG.md