Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add async programming to the consume message in parallel #29

Open
ahmetfurkankavraz opened this issue Dec 20, 2023 · 0 comments
Open

Add async programming to the consume message in parallel #29

ahmetfurkankavraz opened this issue Dec 20, 2023 · 0 comments
Assignees
@ahmetfurkankavraz

Comments

@ahmetfurkankavraz
Copy link
Contributor

We can extract the message consumption section to a async function and then wait all calls outside of the loop with a Task.WhenAll(). This will increase the parallelism and we will decrease total duration of program.
@ahmetfurkankavraz ahmetfurkankavraz self-assigned this May 17, 2024
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue May 17, 2024
Trendyol#29 and Trendyol#30 change the error catching mechanism to to…
bba2d28 
…pic-partition-based and apply topic-based parallel programming to optimize program running time
@ahmetfurkankavraz ahmetfurkankavraz mentioned this issue May 17, 2024
Merged
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue May 19, 2024
Trendyol#29 and Trendyol#44 fixes async programming race condition is…
6c4fa0d 
…sues and limits the message consumption based on topics intead of topic partitions
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue May 19, 2024
Trendyol#29 continues to consume messages if the topic partition lag …
e160087 
…is not cleared yet
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue May 19, 2024
Trendyol#29 increases message consume timeout limit
e60a4c0
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue May 20, 2024
Trendyol#29 prevents to initialize kafka consumers for empty topic pa…
2eecccf 
…rtitions
@ahmetfurkankavraz ahmetfurkankavraz changed the title Add async programming to the consume message part Add async programming to the consume message in parallel May 28, 2024
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue May 28, 2024
Trendyol#29 Trendyol#30 Trendyol#44 update versions in changelog and …
5d99eb1 
…gitlab-ci
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue Jul 29, 2024
Trendyol#29 uses try finally block to avoid deadlock for async calls
a3412ff
MehmetFiratKomurcu added a commit that referenced this issue Aug 29, 2024
@MehmetFiratKomurcu
Merge pull request #43 from ahmetfurkankavraz/main
7084cf5 
#29, #30 and #44 change the error catching and message consumption limit mechanism to topic-partition-based and apply topic-based parallel programming to optimize program running time
ahmetfurkankavraz pushed a commit to ahmetfurkankavraz/kafka-retry-job that referenced this issue Aug 29, 2024
Trendyol#29 adds missing configurations to the read me
648f0d2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ahmetfurkankavraz ahmetfurkankavraz

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ahmetfurkankavraz