-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Increase retry tokens to support streams with lots of shards
We are seeing the quota error during `SubscribeToShard` because Kinesis client is shared across go routines concurrently subscribing to multiple shards. They are all sharing the retry token bucket which has 500 tokens by default. Each retry of `SubscribeToShard` costs 5 tokens. In streams with large number of shards it's possible that more than 100 `SubscribeToShard` calls are failing and therefore runs out of retry tokens. This change increases the token count to 1000000. Once we verify the stability of this fix, we may introduce a mechanism to dynamically workout the token count based on the number of shards in the target. Also includes: - Go version is updated 1.19 - All dependencies are upgraded with go get -u Fixes #7
- Loading branch information
Showing
4 changed files
with
72 additions
and
23 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,36 +1,37 @@ | ||
| module khs | ||
|
|
||
| go 1.18 | ||
| go 1.19 | ||
|
|
||
| require ( | ||
| github.com/aws/aws-sdk-go-v2 v1.16.3 | ||
| github.com/aws/aws-sdk-go-v2/config v1.13.0 | ||
| github.com/aws/aws-sdk-go-v2/service/kinesis v1.13.0 | ||
| github.com/aws/smithy-go v1.11.2 | ||
| github.com/aws/aws-sdk-go-v2 v1.17.6 | ||
| github.com/aws/aws-sdk-go-v2/config v1.18.16 | ||
| github.com/aws/aws-sdk-go-v2/service/kinesis v1.17.7 | ||
| github.com/aws/smithy-go v1.13.5 | ||
| github.com/cheggaaa/pb v1.0.29 | ||
| github.com/fatih/color v1.13.0 | ||
| github.com/fatih/color v1.15.0 | ||
| github.com/google/uuid v1.3.0 | ||
| github.com/stretchr/testify v1.7.1 | ||
| ) | ||
|
|
||
| require ( | ||
| github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.2.0 // indirect | ||
| github.com/aws/aws-sdk-go-v2/credentials v1.8.0 // indirect | ||
| github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.10.0 // indirect | ||
| github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.4 // indirect | ||
| github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.2.0 // indirect | ||
| github.com/aws/aws-sdk-go-v2/internal/ini v1.3.4 // indirect | ||
| github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.7.0 // indirect | ||
| github.com/aws/aws-sdk-go-v2/service/sso v1.9.0 // indirect | ||
| github.com/aws/aws-sdk-go-v2/service/sts v1.14.0 // indirect | ||
| github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect | ||
| github.com/aws/aws-sdk-go-v2/credentials v1.13.16 // indirect | ||
| github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.24 // indirect | ||
| github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.30 // indirect | ||
| github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.24 // indirect | ||
| github.com/aws/aws-sdk-go-v2/internal/ini v1.3.31 // indirect | ||
| github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.24 // indirect | ||
| github.com/aws/aws-sdk-go-v2/service/sso v1.12.5 // indirect | ||
| github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.5 // indirect | ||
| github.com/aws/aws-sdk-go-v2/service/sts v1.18.6 // indirect | ||
| github.com/davecgh/go-spew v1.1.1 // indirect | ||
| github.com/jmespath/go-jmespath v0.4.0 // indirect | ||
| github.com/mattn/go-colorable v0.1.9 // indirect | ||
| github.com/mattn/go-isatty v0.0.14 // indirect | ||
| github.com/mattn/go-runewidth v0.0.13 // indirect | ||
| github.com/mattn/go-colorable v0.1.13 // indirect | ||
| github.com/mattn/go-isatty v0.0.17 // indirect | ||
| github.com/mattn/go-runewidth v0.0.14 // indirect | ||
| github.com/pmezard/go-difflib v1.0.0 // indirect | ||
| github.com/rivo/uniseg v0.2.0 // indirect | ||
| github.com/rivo/uniseg v0.4.4 // indirect | ||
| golang.org/x/exp v0.0.0-20220428152302-39d4317da171 // indirect | ||
| golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect | ||
| golang.org/x/sys v0.6.0 // indirect | ||
| gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c // indirect | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters