Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-2381 : Support kafka parallel-consumer #3161

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

chickenchickenlove
Copy link
Contributor

@chickenchickenlove chickenchickenlove commented Mar 24, 2024

Introduction

Hi, Spring Team.
This is chickenchickenlove, huge fan of you guys.
I saw this issue (Support kafka parallel-consumer), and this comment as well!

No plans currently; but contributions are welcome!

So, i'd like to implement that integrate Parallel Consumer to spring-kafka
This is skeleton code.

Please take a look when you have some free time. 🙇‍♂️
Thank you for your time!

Background

  • Sometimes, Confluent Parallel consumer will be more helpful than kafka consumer, because parallel consumer have good parallel processing (If you want to know more, See this).
  • Thus, i think it will be good option to spring-kafka as well.

How To use

@EnableKafka
@EnableParallelConsumer
@SpringBootApplication
public class SpringkafkasampleApplication {

	public static void main(String[] args) {
		SpringApplication.run(SpringkafkasampleApplication.class, args);
	}
}
  • Annotated @EnableParallelConsumer to your Spring application. then, some spring bean related with parallel consumer, will be registered to ApplicationContext
@Component
public class MyTestClass implements ParallelConsumerCallback<String, String> {

    private final List<String> topics;
    public MyTestClass(List<String> topics) {
        this.topics = topics;
    }

    @Override
    public void accept(PollContext<String, String> context) {
        context.getConsumerRecordsFlattened().forEach(kvConsumerRecord -> System.out.println(kvConsumerRecord));
    }

    @Override
    public List<String> getTopics() {
        return this.topics;
    }
}
  • and then, you should implement interface ParallelConsumerCallback and register their concrete class as spring bean by annotating @Component or @Configuration.
  • then, ParallelConsumerFactory will start ParallelConsumer and ParallelConsumer process records with ParallelConsumerCallback concrete class.

ETC

  • How about follow the way like @KafkaListener?
    • IMHO, it is unnecessary. because AFAIK, ParallelConsumer can allow only one callback. thus, it is not suitable for ParallelConsumer.
  • Can we remove ParallelConsumerImportSelector?
    • Yes. if org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration.java class support to register spring bean, ParallelConsumerImportSelector can be removed.

Update from Apr 27.

  • I added some interfaces for parallelConsumer.

Thank you for your time, Again.

@pivotal-cla
Copy link

@chickenchickenlove Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

@pivotal-cla
Copy link

@chickenchickenlove Thank you for signing the Contributor License Agreement!

@sobychacko
Copy link
Contributor

@chickenchickenlove Thank you for the PR and the kind words. Next month, we will release the first RC version of 3.2.0. Since this is an entirely new feature and involves dependencies from io.confluent (as opposed to org.apache for the regular consumer), this needs a detailed review to come up with a plan for supporting parallel consumers. I think you are in the right direction, but we will have an internal discussion on this matter. Depending on how things go, we may have to defer this feature to the 3.3.x release, but we will let you know.

@chickenchickenlove
Copy link
Contributor Author

Hello, @sobychacko !
Thanks for your time 🙇‍♂️

I see! however, that's sounds great. 👍
If after internal discussions, spring-kafka team decides to introduce ParallelConsumer to spring-kafka, and there is anything more to do, Please let me know. I'm more than willing to help.

Also, Please let me know in case of a decision not to introduce ParallelConsumer to spring-kafka!"

Thanks for your time, again 🙇‍♂️

@sobychacko sobychacko added this to the Backlog milestone Mar 25, 2024
@tagorenathv
Copy link

tagorenathv commented May 7, 2024

Any update on this please ?

@sobychacko
Copy link
Contributor

@tagorenathv We will look at this after the GA for 3.2.0 (this month) as part of the next minor version.

@chickenchickenlove chickenchickenlove changed the title Introduce Confluent ParallelConsumer to Spring Kafka GH-2381 : Support kafka parallel-consumer May 10, 2024
@BartlomiejSzczotka
Copy link

BartlomiejSzczotka commented Jul 1, 2024

hello @sobychacko any plans to implement it in near future? Thank you in advance for info!

@sobychacko
Copy link
Contributor

@BartlomiejSzczotka Yes, we will still look at this for the 3.3.0 version. It may not be possible for it to be ready in time for the upcoming M1 release, but it will likely be in the M2 timeframe. Thanks!

@chickenchickenlove
Copy link
Contributor Author

Hi, @sobychacko !
FYI, this PR is currently a draft because we haven't reached a consensus yet.
This means that it reflects only my perspective.
I believe there are better ways to support ParallelConsumer, and we can find them through discussion.

Also, i made new commits for improving code quality at April 27, 28!

So, Take your times and please feel free to talk anything, any thoughs!
Once we reach a consensus, i will write test codes for this feature as well!

Thanks a lot to look at this PR! 😃

@chickenchickenlove
Copy link
Contributor Author

Hi, @sobychacko .Gently ping. 🙇‍♂️

It's been a while.
Is the integration of the parallel consumer still under review?
Please let me know 🙏

@sobychacko
Copy link
Contributor

@chickenchickenlove For various reasons on our end, we decided to postpone this feature to either the next minor (3.4.0) or the next major release (4.0.0). Either way, we will start working on it in a few months.

@chickenchickenlove
Copy link
Contributor Author

chickenchickenlove commented Sep 30, 2024

Thanks for checking 👍
I will be waiting for cue 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants