Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve][misc] Sync commits from apache into 3.1_ds #311

Merged
merged 3 commits into from
Sep 5, 2024
Merged

[improve][misc] Sync commits from apache into 3.1_ds #311

merged 3 commits into from
Sep 5, 2024

Commits on Sep 5, 2024

  1. [improve][broker] Add msgInReplay subscription stat and metric to imp… 

    …rove Key_Shared observability (apache#23224)

(cherry picked from commit 59424a8)
(cherry picked from commit 766d2a4)
    @lhotari @nikhil-ctds
    lhotari authored and nikhil-ctds committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    0479463 View commit details
    Browse the repository at this point in the history

  2. [improve][client]PIP-359:Support custom message listener executor for… 

    … specific subscription (apache#22861)

Co-authored-by: duanlinlin <[email protected]>
[PIP-359](apache#22902)
Support custom message listener thread pool for specific subscription, avoid individual subscription listener consuming too much time leading to higher consumption delay in other subscriptions.

<!--
### Contribution Checklist

  - PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*.

  - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.

  - Each pull request should address only one issue, not mix up code from multiple issues.

  - Each commit in the pull request has a meaningful commit message

  - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
-->

<!-- Either this PR fixes an issue, -->

<!-- or this PR is one task of an issue -->

<!-- If the PR belongs to a PIP, please add the PIP link here -->

<!-- Details of when a PIP is required and how the PIP process work, please see: https://github.com/apache/pulsar/blob/master/pip/README.md -->

### Motivation
In our scenario, there is a centralized message proxy service, this service will use the same PulsarClient instance to create a lot of subscription groups to consume many topics and cache messages locally.Then the business will pull messages from the cache of the proxy service. It seems that there is no problem, but during use, we found that when the
message processing time of several consumer groups (listener mode) is very high, it almost affects all consumer groups responsible for the proxy service, causing a large number of message delays.

By analyzing the source code, we found that by default, all consumer instances created from the same PulsarClient will share a thread pool to process message listeners, and sometimes there are multiple consumer message listeners bound to the same thread. Obviously, when a consumer processes messages and causes long-term blocking, it will cause the messages of other consumers bound to the thread to fail to be processed in time, resulting in message delays. Therefore, for this scenario, it may be necessary to support specific a message listener thread pool with consumer latitudes to avoid mutual influence between different consumers.

<!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. -->

### Modifications
Support custom message listener thread pool for specific subscription.
<!-- Describe the modifications you've done. -->

(cherry picked from commit 10f4e02)
(cherry picked from commit c5846bb)
    @AuroraTwinkle @nikhil-ctds
    AuroraTwinkle authored and nikhil-ctds committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    a432fcb View commit details
    Browse the repository at this point in the history

  3. [fix][broker] Fix brokers still retry start replication after closed … 

    …the topic (apache#23237)

(cherry picked from commit aee2ee5)
(cherry picked from commit 311b6af)
    @poorbarcode @nikhil-ctds
    poorbarcode authored and nikhil-ctds committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    73b06fa View commit details
    Browse the repository at this point in the history