Binary size too big with s3 packages

[ncw]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

When I add the s3 backend which uses the SDKv2 to rclone it increases the binary by 7.4 MiB and users have been complaining about the large binary sizes.

This makes the s3 backend by far the largest contributor to the rclone binary size (about 13%). Here are the top 11 backends ranked by size measured by commenting the backend out and compiling in release mode to check the binary sizes.

Backend	Size MiB
s3	7.43
storj	4.24
protondrive	2.61
hdfs	2.41
oracleobjectstorage	1.60
drive	1.53
filescom	1.46
azureblob	1.26
dropbox	1.15
googlecloudstorage	0.78
azurefiles	0.65

I compared this to the last version of rclone which used the SDKv1 and the s3 backend takes 6.9 MiB so this has got slightly worse. I expected this to get better with the modularization of the SDK :-(

Rclone uses the following sdk imports

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/aws/signer/v4"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
	"github.com/aws/smithy-go"
	"github.com/aws/smithy-go/logging"
	"github.com/aws/smithy-go/middleware"
	"github.com/aws/smithy-go/transport/http"

According to the rather neat tool go-size-analyzer

The bulk of this in the service modules

Is there any way this can be improved?

Thank you

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I expected the S3 packages from the SDK to add no more than 1MB to my binary

Current Behavior

It adds 7.5 MB to my binary

Reproduction Steps

The test_backend_sizes.py script will compile rclone many times to measure backend sizes, and the go-size-analyzer tool can be used to verify it by running it on the rclone binary.

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

go mod graph as attachment as it is quite big.

graph.txt

Compiler and Version used

go version go1.23.4 linux/amd64

Operating System and version

Ubuntu 22.04.5 LTS

[RanVaknin]

Hi @ncw,

Thanks for reaching out. S3 is indeed a large client because it is one of the biggest AWS services in terms of its API representation size.
We can do more about trying to reduce the binary size but it will likely happen as a "side effect" of other efforts, for example #2933.

I'm going to keep this is a backlog item to gauge community interest.

Thanks,
Ran~