Skip to content
This repository has been archived by the owner on Aug 31, 2023. It is now read-only.

aws s3 sync is slow #312

Open
Atry opened this issue Jul 21, 2022 · 2 comments
Open

aws s3 sync is slow #312

Atry opened this issue Jul 21, 2022 · 2 comments

Comments

@Atry
Copy link
Contributor

Atry commented Jul 21, 2022

We use aws s3 sync to upload deb packages from EBS to S3

aws s3 sync "$REPO_DIR/" "s3://hhvm-downloads/${DISTRO_BASE}/" --exclude '*' --include '*.deb' --include '*.orig.tar.gz'
aws s3 sync "$REPO_DIR/" "s3://hhvm-downloads/${DISTRO_BASE}/"

According to the document, aws s3 sync will compare the content of all the 2TB files even when we just added 5GB files. It would be good to avoid checking the 2TB files.

@Atry
Copy link
Contributor Author

Atry commented Jul 21, 2022

Even though it is still not ideal, I think Fred had partially optimized it, because previously we had to downloading the 2TB files before the updating, which is worse.

@Atry
Copy link
Contributor Author

Atry commented Jul 21, 2022

Why not mount S3 with s3fs and directly run reprepro in the mounted path, instead of aws s3 sync?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant