Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIX] - #9011 | Solution for AWS CLI slow startup #1

Merged
merged 6 commits into from
Nov 2, 2024

Conversation

imSanko
Copy link
Owner

@imSanko imSanko commented Nov 2, 2024

Issue aws#9011, if available: AWS CLI slow startup

Description of changes:
The shell script that includes optimizations for the AWS CLI to improve its performance on local S3-compatible services. This script sets the recommended configurations and runs the optimized aws s3 command can be done:

  1. AWS_EC2_METADATA_DISABLED=true: Disables metadata calls to EC2, which aren’t needed for a local S3-compatible instance.
  2. max_concurrent_requests and max_queue_size: Configures S3 to handle more requests concurrently, which can improve throughput.
  3. cli_connect_timeout and cli_read_timeout: Reduces timeout values to prevent delays in local connections.
  4. --no-paginate: Disables pagination, reducing the overhead for commands that list objects in the bucket.
#!/bin/bash


AWS_PROFILE="s3proxy"
AWS_REGION="us-east-1"
ENDPOINT_URL="http://127.0.0.1:8081"
BUCKET_NAME="s3://your-bucket-name"

aws configure set profile.$AWS_PROFILE.s3.max_concurrent_requests 10
aws configure set profile.$AWS_PROFILE.s3.max_queue_size 1000
aws configure set profile.$AWS_PROFILE.s3.max_bandwidth 50MB
aws configure set profile.$AWS_PROFILE.s3.use_accelerate_endpoint false
aws configure set default.cli_connect_timeout 5
aws configure set default.cli_read_timeout 10

echo "Listing S3 bucket contents with AWS CLI..."
time AWS_EC2_METADATA_DISABLED=true aws s3 ls "$BUCKET_NAME" \
    --region "$AWS_REGION" \
    --endpoint-url "$ENDPOINT_URL" \
    --profile "$AWS_PROFILE" \
    --no-paginate

echo "Done."

Add On's:

This allows for TCP reuse, reducing the time delay from creating new connections, which is helpful in local environments. Make sure to replace s3://your-bucket-name with your actual bucket name.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@imSanko imSanko added bug Something isn't working enhancement New feature or request labels Nov 2, 2024
@imSanko imSanko self-assigned this Nov 2, 2024
@imSanko imSanko merged commit d5fa3db into imSanko:develop Nov 2, 2024
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants