v3.3.3

2024-12-03

Library updates

• [fix][sec] Drop hdfs2 support, Upgrade hadoop3 to 3.4.0 and dnsjava to 3.6.2 to address CVE-2024-25638 (#23411)
• [fix][sec] Replace bcprov-jdk15on dependency with bcprov-jdk18-on (#23532)
• [fix][sec] Upgrade Zookeeper to 3.9.3 to address CVE-2024-51504 (#23581)
• [fix][sec] Upgrade to Netty 4.1.115.Final to address CVE-2024-47535 (#23596)
• [improve] Upgrade OpenTelemetry library to 1.44.1 version (#23656)
• [improve] Upgrade oxia-java to 0.4.10 and fix closing of OxiaMetadataStore (#23653)

Broker

• [fix][broker] Avoid orphan ledgers in BucketDelayedDeliveryTracker (#22802)
• [fix][broker] Broker is failing to create non-durable sub if topic is fenced (#23579)
• [fix][broker] Fix AvgShedder strategy check (#23156)
• [fix][broker] Fix Broker migration NPE while broker tls url not configured (#23534)
• [fix][broker] Fix currently client retries until operation timeout if the topic does not exist (#23530)
• [fix][broker] Fix failed TokenAuthenticatedProducerConsumerTest (#23602)
• [fix][broker] Fix namespace unload might be blocked too long with extensible load manager (#23433) (#23517)
• [fix][broker] Fix ownership loss (#23515)
• [fix][broker] Fix print cluster migration state response (#23535)
• [fix][broker] Fix race-condition causing repeated delete topic (#23522)
• [fix][broker] Fix the broker registering might be blocked for long time (#23371) (#23507)
• [fix][broker] Fix unloadNamespaceBundlesGracefully can be stuck with extensible load manager (#23349) (#23496)
• [fix][broker] Increase readBuffer size for bookkeeper.DLOutputStream (#23548)
• [fix][broker] Topics failed to delete after remove cluster from replicated clusters set and caused OOM (#23360)
• [fix][broker] fix logging with correct error message while loading the topic (#23544)
• [fix][broker] fix null lookup result when brokers are starting (#23642)
• [fix][broker] normalize path (#23438)
• [fix][broker] timeout when broker registry hangs and monitor broker registry (ExtensibleLoadManagerImpl only) (#23382) (#23510)
• [improve][broker] Add log to track issue when handleGetTopicsOfNamespace (#23434)
• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
• [improve][broker] Decouple pulsar_storage_backlog_age_seconds metric with backlogQuota check (#23619)
• [improve][broker] Exclude system topics from namespace level publish and dispatch rate limiting (#23589)
• [improve][broker] Make cluster metadata init command support metadata config path (#23269)
• [improve][broker] Make cluster metadata teardown command support metadata config path (#23520)
• [improve][broker] PIP-383: Support granting/revoking permissions for multiple topics (#23372)
• [improve][broker] PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic (#23584)
• [improve][broker] Skip unloading when bundle throughput is zero (ExtensibleLoadManagerImpl only) (#23626)
• [improve][broker] Support cleanup replication cluster and allowed cluster when cluster metadata teardown (#23561)
• [improve][broker] re-elect the channel owner if no channel owner is found (#23516) (#23580)
• [improve][broker] replace HashMap with inner implementation ConcurrentLongLongPairHashMap in Negative Ack Tracker. (#23582)

Client

• [fix][admin] Fix lookup get a null result if uses proxy (#23556)
• [fix][client] Fix DLQ producer name conflicts when there are same name consumers (#23577)
• [fix][client] Fix Reader.hasMessageAvailable return wrong value after seeking by timestamp with startMessageIdInclusive (#23502)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix deadlock of NegativeAcksTracker (#23651)
• [fix][client] Fix producer/consumer stop to reconnect or Pub/Sub due to IO thread race-condition (#23499)
• [fix][client] Fix race-condition causing doReconsumeLater to hang when creating retryLetterProducer has failed (#23560)
• [fix][client] Fix the javadoc for ConsumerBuilder.isAckReceiptEnabled (#23452)
• [fix][client] Fixed an issue where a cert chain could not be used in TLS authentication (#23644)
• [fix][client] Initializing client-authentication using configured auth params (#23610)
• [fix][client] Make protobuf-java dependency optional in java client libraries (#23632)
• [fix][client] Prevent embedding protobuf-java class files in pulsar-client-admin and pulsar-client-all (#23468)
• [fix][client] The partitionedProducer maxPendingMessages always is 0 (#23593)
• [fix][client] Use dedicated executor for requests in BinaryProtoLookupService (#23378)
• [fix][client] fix incomingMessageSize and client memory usage is negative (#23624)
• [fix][client] fix the beforeConsume() method earlier hit with message listener (#23578)
• [improve][admin] Print error log if handle http response fails (#23563)
• [improve][client] Enhance error handling for non-exist subscription in consumer creation (#23254)
• [improve][client] Increase default Java client connectionMaxIdleSeconds to 60 seconds (#23430)
• [improve][client] Reduce unshaded dependencies and shading warnings in shaded Java client modules (#23647)
• [improve][client] Replace NameUtil#generateRandomName with RandomStringUtils#randomAlphanumeric (#23645)

Pulsar IO and Pulsar Functions

• [fix][fn] ack messages for window function when its result is null (#23618)
• [improve][io] Support update subscription position for sink connector (#23538)
• [improve][io] Upgrade Spring version to 6.1.13 in IO Connectors (#23459)

Others

• [fix][misc] Class conflict during jetcd-core-shaded shading process (#23641)
• [fix][misc] Unable to connect an etcd metastore with recent releases due to jetc-core sharding problem (#23604)
• [fix][ml] Managed ledger should recover after open ledger failed (#23368)
• [fix][proxy] Fix pattern consumer does not work when using Proxy (#23489)
• [fix][ws] Implement missing http header data functions in AuthenticationDataSubscription (#23638)
• [improve] Improve logic for enabling Netty leak detection (#23613)
• [improve] Use single buffer for metrics when noUnsafe use (#23612)
• [improve][misc] Disable OTel by default when running the pulsar-perf tool (#23585)
• [improve][misc] Upgrade Jetty to 9.4.56.v20240826 (#23405)
• [improve][ml] Avoid repetitive nested lock for isMessageDeleted in ManagedCursorImpl ([#23609](htt...

v4.0.1

2024-12-02

Library updates

• [feat][meta] Bump oxia java version from 0.4.5 to 0.4.7 (#23471)
• [feat][misc] Upgrade oxia version to 0.4.9 (#23607)
• [fix][sec] Replace bcprov-jdk15on dependency with bcprov-jdk18-on (#23532)
• [fix][sec] Upgrade Zookeeper to 3.9.3 to address CVE-2024-51504 (#23581)
• [fix][sec] Upgrade to Netty 4.1.115.Final to address CVE-2024-47535 (#23596)
• [improve][monitor] Upgrade OTel to 1.41.0 (#23484)
• [improve] Upgrade OpenTelemetry library to 1.44.1 version (#23656)
• [improve] Upgrade oxia-java to 0.4.10 and fix closing of OxiaMetadataStore (#23653)

Broker

• [feat][broker] Implement allowBrokerOperationAsync in PulsarAuthorizationProvider to avoid exception thrown (#23663)
• [fix][broker] Broker is failing to create non-durable sub if topic is fenced (#23579)
• [fix][broker] Fix Broker migration NPE while broker tls url not configured (#23534)
• [fix][broker] Fix currently client retries until operation timeout if the topic does not exist (#23530)
• [fix][broker] Fix failed TokenAuthenticatedProducerConsumerTest (#23602)
• [fix][broker] Fix ownership loss (#23515)
• [fix][broker] Fix print cluster migration state response (#23535)
• [fix][broker] Fix race-condition causing repeated delete topic (#23522)
• [fix][broker] Increase readBuffer size for bookkeeper.DLOutputStream (#23548)
• [fix][broker] Key_Shared subscription: Reject consumers with incompatible policy (#23449)
• [fix][broker] Revert "[improve][client] Add log when can't add message to the container (#23657)
• [fix][broker] fix logging with correct error message while loading the topic (#23544)
• [fix][broker] fix null lookup result when brokers are starting (#23642)
• [fix][broker] support missing tenant level fine-granted permissions (#23660)
• [fix][broker]: support missing broker level fine-granted permissions (#23637)
• [improve][broker] Decouple pulsar_storage_backlog_age_seconds metric with backlogQuota check (#23619)
• [improve][broker] Exclude system topics from namespace level publish and dispatch rate limiting (#23589)
• [improve][broker] Make cluster metadata teardown command support metadata config path (#23520)
• [improve][broker] PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic (#23584)
• [improve][broker] Reduce memory occupation of InMemoryRedeliveryTracker. (#23640)
• [improve][broker] Reduce memory occupation of the delayed message queue (#23611)
• [improve][broker] Skip unloading when bundle throughput is zero (ExtensibleLoadManagerImpl only) (#23626)
• [improve][broker] Support cleanup replication cluster and allowed cluster when cluster metadata teardown (#23561)
• [improve][broker] re-elect the channel owner if no channel owner is found (#23516)
• [improve][broker] replace HashMap with inner implementation ConcurrentLongLongPairHashMap in Negative Ack Tracker. (#23582)

Client

• [fix][admin] Fix lookup get a null result if uses proxy (#23556)
• [fix][client] Fix ConsumerStats.getRateMsgsReceived javadoc (#21618)
• [fix][client] Fix DLQ producer name conflicts when there are same name consumers (#23577)
• [fix][client] Fix Reader.hasMessageAvailable return wrong value after seeking by timestamp with startMessageIdInclusive (#23502)
• [fix][client] Fix deadlock of NegativeAcksTracker (#23651)
• [fix][client] Fix producer/consumer stop to reconnect or Pub/Sub due to IO thread race-condition (#23499)
• [fix][client] Fix race-condition causing doReconsumeLater to hang when creating retryLetterProducer has failed (#23560)
• [fix][client] Fix the javadoc for startMessageIdInclusive (#23508)
• [fix][client] Fixed an issue where a cert chain could not be used in TLS authentication (#23644)
• [fix][client] Initializing client-authentication using configured auth params (#23610)
• [fix][client] Make protobuf-java dependency optional in java client libraries (#23632)
• [fix][client] Prevent embedding protobuf-java class files in pulsar-client-admin and pulsar-client-all (#23468)
• [fix][client] The partitionedProducer maxPendingMessages always is 0 (#23593)
• [fix][client] Use dedicated executor for requests in BinaryProtoLookupService (#23378)
• [fix][client] fix incomingMessageSize and client memory usage is negative (#23624)
• [fix][client] fix the beforeConsume() method earlier hit with message listener (#23578)
• [improve][admin] Print error log if handle http response fails (#23563)
• [improve][client] Enhance error handling for non-exist subscription in consumer creation (#23254)
• [improve][client] Reduce unshaded dependencies and shading warnings in shaded Java client modules (#23647)
• [improve][client] Replace NameUtil#generateRandomName with RandomStringUtils#randomAlphanumeric (#23645)

Pulsar IO and Pulsar Functions

• [fix][fn] ack messages for window function when its result is null (#23618)
• [improve][io] Support update subscription position for sink connector (#23538)
• [improve][io] Upgrade Spring version to 6.1.13 in IO Connectors (#23459)
• [improve][io] Upgrade Spring version to 6.1.14 in IO Connectors (#23481)

Others

• [feat][monitor] Add offloader stats grafana dashboard (#23479)
• [fix][misc] Class conflict during jetcd-core-shaded shading process (#23641)
• [fix][misc] Unable to connect an etcd metastore with recent releases due to jetc-core sharding problem (#23604)
• [fix][proxy] Fix pattern consumer does not work when using Proxy (#23489)
• [fix][standalone] correctly delete bookie registration znode (#23497)
• [fix][ws] Implement missing http header data functions in AuthenticationDataSubscription (#23638)
• [improve] Improve logic for enabling Netty leak detection (#23613)
• [improve] Use single buffer for metrics when noUnsafe use (#23612)
• [improve][misc] Disable OTel by default when running the pulsar-perf tool (#23585)
• [improve][ml] Avoid repetitive nested lock for isMessageDeleted in ManagedCursorImpl (#23609)
• [improve][offload] Use filesystemURI as the storage path (#23591)
• Enabling DNS retryOnTimeout with TCP in DnsNameResolver (#23590)

Tests & CI

• [cleanup][build] skip generating pom.xml.versionsBackup (#23639)
• [fix][build] Fix error "Element encoding is not allowed here" in pom.xml (#23655)
• [fix][test] Fix DeadLetterTopicTest.testDeadLetterTopicWithInitialSubscriptionAndMultiConsumers (#23552)
• [fix][test] Fix ManagedCursorTest.testForceCursorRecovery (#23518)
• [fix][test] Fix SimpleProducerConsumerT...

v3.0.8

2024-12-02

Library updates

• [fix][sec] Upgrade Zookeeper to 3.9.3 to address CVE-2024-51504 (#23581)
• [fix][sec] Upgrade to Netty 4.1.115.Final to address CVE-2024-47535 (#23596)
• [improve][misc] Upgrade Jetty to 9.4.56.v20240826 (#23405)

Broker

• [fix][broker] Avoid orphan ledgers in BucketDelayedDeliveryTracker (#22802)
• [fix][broker] Broker is failing to create non-durable sub if topic is fenced (#23579)
• [fix][broker] Continue closing even when executor is shut down (#22599)
• [fix][broker] Fix AvgShedder strategy check (#23156)
• [fix][broker] Fix currently client retries until operation timeout if the topic does not exist (#23530)
• [fix][broker] Fix failed TokenAuthenticatedProducerConsumerTest (#23602)
• [fix][broker] Fix ownership loss (#23515)
• [fix][broker] Fix race-condition causing repeated delete topic (#23522)
• [fix][broker] Increase readBuffer size for bookkeeper.DLOutputStream (#23548)
• [fix][broker] Topics failed to delete after remove cluster from replicated clusters set and caused OOM (#23360)
• [fix][broker] fix null lookup result when brokers are starting (#23642)
• [fix][broker] normalize path (#23438)
• [improve][broker] Add log to track issue when handleGetTopicsOfNamespace (#23434)
• [improve][broker] Close TopicPoliciesService to allow Pulsar broker graceful shutdown (#22589)
• [improve][broker] Decouple pulsar_storage_backlog_age_seconds metric with backlogQuota check (#23619)
• [improve][broker] Don't use forkjoin pool by default for deleting partitioned topics (#22598)
• [improve][broker] Exclude system topics from namespace level publish and dispatch rate limiting (#23589)
• [improve][broker] Make cluster metadata init command support metadata config path (#23269)
• [improve][broker] Make cluster metadata teardown command support metadata config path (#23520)
• [improve][broker] PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic (#23584)
• [improve][broker] Support cleanup replication cluster and allowed cluster when cluster metadata teardown (#23561)
• [improve][broker] replace HashMap with inner implementation ConcurrentLongLongPairHashMap in Negative Ack Tracker. (#23582)

Client

• [fix][admin] Fix lookup get a null result if uses proxy (#23556)
• [fix][client] Fix Reader.hasMessageAvailable return wrong value after seeking by timestamp with startMessageIdInclusive (#23502)
• [fix][client] Fix deadlock of NegativeAcksTracker (#23651)
• [fix][client] Fix producer/consumer stop to reconnect or Pub/Sub due to IO thread race-condition (#23499)
• [fix][client] Fix race-condition causing doReconsumeLater to hang when creating retryLetterProducer has failed (#23560)
• [fix][client] Fix the javadoc for ConsumerBuilder.isAckReceiptEnabled (#23452)
• [fix][client] Fixed an issue where a cert chain could not be used in TLS authentication (#23644)
• [fix][client] Initializing client-authentication using configured auth params (#23610)
• [fix][client] Make protobuf-java dependency optional in java client libraries (#23632)
• [fix][client] Prevent embedding protobuf-java class files in pulsar-client-admin and pulsar-client-all (#23468)
• [fix][client] The partitionedProducer maxPendingMessages always is 0 (#23593)
• [fix][client] Use dedicated executor for requests in BinaryProtoLookupService (#23378) (#23461)
• [fix][client] fix incomingMessageSize and client memory usage is negative (#23624)
• [fix][client] fix the beforeConsume() method earlier hit with message listener (#23578)
• [improve][admin] Print error log if handle http response fails (#23563)
• [improve][client] Enhance error handling for non-exist subscription in consumer creation (#23254)
• [improve][client] Increase default Java client connectionMaxIdleSeconds to 60 seconds (#23430)
• [improve][client] Reduce unshaded dependencies and shading warnings in shaded Java client modules (#23647)

Pulsar IO and Pulsar Functions

• [fix][fn] ack messages for window function when its result is null (#23618)
• [fix][sql][branch-3.0] Fix long decimal compatibility in Trino 368. (#23419)
• [fix][sql][branch-3.0] Fix shading configuration for presto-pulsar
• [improve][io] Support update subscription position for sink connector (#23538)
• [improve][io] Upgrade Spring version to 6.1.13 in IO Connectors (#23459)

Others

• [fix] Restored method as deprecated in AbstractMetadataStore (#21950)
• [fix][misc] Class conflict during jetcd-core-shaded shading process (#23641)
• [fix][misc] Unable to connect an etcd metastore with recent releases due to jetc-core sharding problem (#23604)
• [fix][ml] Managed ledger should recover after open ledger failed (#23368)
• [fix][proxy] Fix pattern consumer does not work when using Proxy (#23489)
• [fix][standalone] correctly delete bookie registration znode (#23497)
• [improve] Improve logic for enabling Netty leak detection (#23613)
• [improve] Use single buffer for metrics when noUnsafe use (#23612)
• [fix][ws] Implement missing http header data functions in AuthenticationDataSubscription (#23638)
• [improve][ml] Avoid repetitive nested lock for isMessageDeleted in ManagedCursorImpl (#23609)
• [improve][offload] Use filesystemURI as the storage path (#23591)
• Enabling DNS retryOnTimeout with TCP in DnsNameResolver (#23590)

Tests & CI

• [cleanup][build] skip generating pom.xml.versionsBackup (#23639)
• [fix][build] Add basic support for vscode-java and Eclipse IDE (#23448)
• [fix][build] Fix error "Element encoding is not allowed here" in pom.xml (#23655)
• [fix][test] Address flaky GetPartitionMetadataMultiBrokerTest (#23456)
• [fix][test] Fix DeadLetterTopicTest.testDeadLetterTopicWithInitialSubscriptionAndMultiConsumers (#23552)
• [fix][test] Fix SimpleProducerConsumerTest.testMultiTopicsConsumerImplPauseForManualSubscription (#23546)
• [fix][test] Fix flaky GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic (#23259)
• [fix][test] Fix flaky test ManagedLedgerTest.testDeleteCurrentLedgerWhenItIsClosed (#23437)
• [fix][test][branch-3.0] Fix PrecisTopicPublishRateThrottleTest that broke after #23589 changes
• [fix][test][branch-3.0] Fix cherry-picking issue in 2f6c1a3 where Cleanup import was missing
• [improve][build] Require Java 17 for building Pulsar branch-3.0 (#22875)
• [improve][build] Update maven-wrapper (mvnw) to recent stable version 3.3.2 (#23410)
• [improve][build][branch-3.0] Upgrade docker-maven-plugin to 0.45.1
• [improve][test] Added message properties tests for batch and non-batch...

v4.0.0

2024-10-21

Changes in 4.0.0 release since 3.3.0 release preparation was started.
For the complete list, check the full changelog and full PR list.

What's New in Apache Pulsar 4.0? covers topics such as:

• Enhanced Key_Shared subscription implementation
• Secure Docker runtime based on Alpine Linux and Java 21
• Enhanced Quality of Service controls for multi-tenant deployments
• Rate limiting as a foundation for multi-tenant resource management
• Upgrade paths from previous versions
• Client compatibility and Importance of upgrading Java clients to address CVE-2024-47561

PIPs accepted since 3.0.0 release

PIPs accepted in 4.0.0 release

• PIP-327: Support force topic loading for unrecoverable errors (#21752)
• PIP-337: SSL Factory Plugin to customize SSL Context and SSL Engine generation (#22016)
• PIP-347: add role field in consumer's stat (#22564)
• PIP-348: Trigger offload on topic load stage (#22650)
• PIP-349: Add additionalSystemCursorNames ignore list for TTL check (#22651)
• PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis (#22688)
• PIP-351: Additional options for Pulsar-Test client to support KeyStore based TLS (#22694)
• PIP-352: Event time based topic compactor (#22710)
• PIP-353: Improve transaction message visibility for peek-messages (#22746)
• PIP-354: apply topK mechanism to ModularLoadManagerImpl (#22765)
• PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22778)
• PIP-356: Support Geo-Replication starts at earliest position (#22806)
• PIP-357: Correct the conf name in load balance module. (#22823)
• PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. (#22889)
• PIP-359: Support custom message listener executor for specific subscription (#22902)
• PIP-360: Add admin API to display Schema metadata (#22913)
• PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete. (#22940)
• PIP-364: Introduce a new load balance algorithm AvgShedder (#22946)
• PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23033)
• PIP-367: Propose a Contributor Repository for Pulsar (#23061)
• PIP-368: Support lookup based on the lookup properties (#23075)
• PIP-369: Flag based selective unload on changing ns-isolation-policy (#23116)
• PIP-370: configurable remote topic creation in geo-replication (#23124)
• PIP-374: Visibility of messages in receiverQueue for the consumers (#23235)
• PIP-376: Make topic policies service pluggable (#23248)
• PIP-378: Add ServiceUnitStateTableView abstraction (ExtensibleLoadMangerImpl only) (#23300)
• PIP-379: Key_Shared Draining Hashes for Improved Message Ordering (#23309)
  • Documentation: Preserving order of message delivery by key
• PIP-381: Handle large PositionInfo state (#23328)
• PIP-383: Support granting/revoking permissions for multiple topics (#23355)
• PIP-384: ManagedLedger interface decoupling (#23363)

PIPs accepted in 3.3.0 release

• PIP-315: Configurable max delay limit for delayed delivery (#21798)
• PIP-321: Introduce allowed-cluster at the namespace level (#21648)
• PIP-324: Alpine Docker images (#21716)
• PIP-325: Add command to abort transaction (#21731)
• PIP-326: Create a BOM to ease dependency management (#21747)
• PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
• PIP-330: getMessagesById gets all messages (#21873)
• PIP-342: Support OpenTelemetry metrics in Pulsar client (#22178)
• PIP-343: Use picocli instead of jcommander (#22181)
• PIP-344: Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
• PIP-335: Oxia metadata plugin (#22009)
• PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)
• PIP-282: Change definition of the recently joined consumers position (#20776)

PIPs accepted in 3.2.0 release

• PIP-323: Complete Backlog Quota Telemetry (#21709)
• PIP-322: Pulsar Rate Limiting Refactoring (#21680)
• PIP-320: OpenTelemetry Scaffolding (#21635)
• PIP-318: Don't retain null-key messages during topic compaction (#21541)
• PIP-315: Configurable max delay limit for delayed del...

v3.3.2

2024-10-04

Library updates

• [fix][sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561 (#23394)
• [fix][sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391 (#23338)
• [fix][sec][branch-3.3] Upgrade protobuf-java to 3.25.5 (#23356) (#23358)
• [improve][misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66 (#23255)
• [fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0 (#23393)
• [fix] Bump io.grpc from 1.56.0 to 1.56.1 (#23276)
• [fix][metadata] Upgrade Oxia to 0.3.2 (#23140)
• [feat][meta] Bump oxia java version from 0.3.2 to 0.4.5 (#23277)
• [improve] Upgrade Pulsar Python client in docker image to 3.5.0 (#23377)
• [improve] Install openssl in the docker image to fix compatibility with Apache Pulsar Helm chart (#23362)

Broker

• [fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)
• [fix][broker] Execute the pending callbacks in order before ready for incoming requests (#23266)
• [fix][broker] Fail fast if the extensible load manager failed to start (#23297)
• [fix][broker] Fix 'Disabled replicated subscriptions controller' logic and logging (#23142)
• [fix][broker] Fix authenticate order in AuthenticationProviderList (#23111)
• [fix][broker] Fix brokers still retry start replication after closed the topic (#23237)
• [fix][broker] Fix compatibility issues for PIP-344 (#23136)
• [fix][broker] Fix incomplete NAR file extraction which prevents broker from starting (#23274)
• [fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector (#23327)
• [fix][broker] Fix shadow topics cannot be consumed when the entry is not cached (#23147)
• [fix][broker] Fix system topic can not be loaded up if it contains data offloaded (#23279)
• [fix][broker] Fix the broker registery cannot recover from the metadata node deletion (#23359)
• [fix][broker] Fix the bug that elected leader thinks it's a follower (#23138)
• [fix][broker] Handle the case when getOwnedServiceUnits fails gracefully (#23119)
• [fix][broker] Internal reader of __change_events can not started after metadata store session rebuilt (#23018)
• [fix][broker] Let Pending ack handler can retry to init when encounters a metadata store error (#23153)
• [fix][broker] Skip reading entries from closed cursor. (#22751)
• [fix][broker] Topic can never be loaded up due to broker maintains a failed topic creation future (#23184)
• [fix][broker] fix exception may hidden and result in stuck when topic loading (#23102)
• [fix][broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock (#23258)
• [fix][broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync (#23100)
• [fix][broker] type cast on exceptions in exceptionally can lead to lost calls (#23117)
• [fix][broker]A failed consumer/producer future in ServerCnx can never be removed (#23123)
• [improve][broker] Add msgInReplay subscription stat and metric to improve Key_Shared observability (#23224)
• [improve][broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only) (#23230)
• [improve][broker] Avoid subscription fenced error with consumer.seek whenever possible (#23163)
• [improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only) (#23144)
• [improve][broker] Improve pulsar_topic_load_failed metric to record correct failed time (#23199)
• [improve][broker] Optimize high CPU usage when consuming from topics with ongoing txn (#23189)
• [improve][broker] Optimize message payload traffic for ShadowReplicator (#23236)
• [improve][broker] Optimize performance for checking max topics when the topic is a system topic (#23185)
• [improve][broker] Optimize the performance of individual acknowledgments (#23072)
• [improve][broker] Register the broker to metadata store without version id compare (#23298)
• [improve][broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only) (#23152)
• [improve][broker] Support customized shadow managed ledger implementation (#23179)
• [improve][broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata (#23087)
• [improve][broker] Reuse method getAvailableBrokersAsync (#23099)
• [improve][admin] PIP-369 Introduce unload flag in ns-isolation-policy set call (#23120)
• [fix] StatsOutputStream: add string write function (#23227)

Client

• [fix] DLQ to handle bytes key properly (#23172)
• [fix][client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying (#23182)
• [fix][client] Create the retry producer async (#23157)
• [fix][client] Fix for early hit beforeConsume for MultiTopicConsumer (#23141)
• [fix][client] Fix timeout handling in Pulsar Admin client (#23128)
• [fix][client] TransactionCoordinatorClient support retry (#23081)
• [fix][client] the nullValue in msgMetadata should be true by default (#22372)
• [improve][client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder (#22541)
• [improve][client] Don't print info logs for each schema loaded by client (#23206)
• [improve][client] Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover (#23129)

Pulsar IO and Pulsar Functions

• [fix][io] Upgrade mssql server docker tag in DebeziumMsSqlContainer (#23318)
• [improve][fn] Add support for overriding additionalJavaRuntimeArguments with PF_additionalJavaRuntimeArguments env (#23130)

Others

• [fix][log] Do not print error log if tenant/namespace does not exist when calling get topic metadata (#23291)
• [fix][log] Do not print warn log when concurrently publishing and switching ledgers (#23209)
• [feat] Add scripts for updating BK RocksDB ini files (#23178)
• [fix][meta] Oxia metadta store: Convert error to MetadataStoreException if operation failed (#23154)
• [fix][misc] Log Conscrypt security provider initialization warnings at debug level (#23364)
• [improve][misc] Improve AES-GCM cipher performance (#23122)
• [improve][misc] Optimize TLS performance by omitting extra buffer copies (#23115)
• [improve][pip] PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23041)
• [improve][proxy] Reuse authentication instance in pulsar-proxy (#23113)

Tests & CI

• [fix][test] Fix flaky SubscriptionSeekTest.testSeekIsByReceive (#23170)
• [fix][test] Fix flaky UnloadSubscriptionTest.testMultiConsumer ([#23243](#23243...

v3.0.7

2024-10-04

Library updates

• [fix][sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561 (#23394)
• [fix][sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391 (#23338)
• [fix][sec][branch-3.0] Upgrade protobuf-java to 3.25.5 (#23356) (#23357)
• [fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0 (#23393)
• [improve][misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66 (#23255)
• [improve][misc] Upgrade jersey to 2.41 (#22290)
• [improve] Upgrade Pulsar Python client in docker image to 3.5.0 (#23377)

Broker

• [fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)
• [fix][broker] Execute the pending callbacks in order before ready for incoming requests (#23266)
• [fix][broker] Fix 'Disabled replicated subscriptions controller' logic and logging (#23142)
• [fix][broker] Fix authenticate order in AuthenticationProviderList (#23111)
• [fix][broker] Fix brokers still retry start replication after closed the topic (#23237)
• [fix][broker] Fix compatibility issues for PIP-344 (#23136)
• [fix][broker] Fix incomplete NAR file extraction which prevents broker from starting (#23274)
• [fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector (#23327)
• [fix][broker] Fix shadow topics cannot be consumed when the entry is not cached (#23147)
• [fix][broker] Fix system topic can not be loaded up if it contains data offloaded (#23279)
• [fix][broker] Fix the bug that elected leader thinks it's a follower (#23138)
• [fix][broker] Handle the case when getOwnedServiceUnits fails gracefully (#23119)
• [fix][broker] Internal reader of __change_events can not started after metadata store session rebuilt (#23018)
• [fix][broker] Let Pending ack handler can retry to init when encounters a metadata store error (#23153)
• [fix][broker] Skip reading entries from closed cursor. (#22751)
• [fix][broker] Topic can never be loaded up due to broker maintains a failed topic creation future (#23184)
• [fix][broker] fix exception may hidden and result in stuck when topic loading (#23102)
• [fix][broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock (#23258)
• [fix][broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync (#23100)
• [fix][broker] fix the log format error in the AdminResource (#23278)
• [fix][broker] type cast on exceptions in exceptionally can lead to lost calls (#23117)
• [fix][broker]A failed consumer/producer future in ServerCnx can never be removed (#23123)
• [fix][broker][branch-3.0] Fail fast if the extensible load manager failed to start (#23297) (#23302)
• [fix][broker] StatsOutputStream: add string write function (#23227)
• [improve][broker] Add msgInReplay subscription stat and metric to improve Key_Shared observability (#23224)
• [improve][broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only) (#23230)
• [improve][broker] Avoid subscription fenced error with consumer.seek whenever possible (#23163)
• [improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only) (#23144)
• [improve][broker] Improve pulsar_topic_load_failed metric to record correct failed time (#23199)
• [improve][broker] Optimize high CPU usage when consuming from topics with ongoing txn (#23189)
• [improve][broker] Optimize performance for checking max topics when the topic is a system topic (#23185)
• [improve][broker] Optimize the performance of individual acknowledgments (#23072)
• [improve][broker] Part 2 of PIP-370: add metrics "pulsar_replication_disconnected_count" (#23213)
• [improve][broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication (#23169)
• [improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
• [improve][broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only) (#23152)
• [improve][broker] Support customized shadow managed ledger implementation (#23179)
• [improve][broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata (#23087) (#23126)
• [improve][broker] Reuse method getAvailableBrokersAsync (#23099)
• [improve][admin] PIP-369 Introduce unload flag in ns-isolation-policy set call (#23120)

Client

• [fix] DLQ to handle bytes key properly (#23172)
• [fix][client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying (#23182)
• [fix][client] Create the retry producer async (#23157)
• [fix][client] Fix for early hit beforeConsume for MultiTopicConsumer (#23141)
• [fix][client] Fix timeout handling in Pulsar Admin client (#23128)
• [fix][client] TransactionCoordinatorClient support retry (#23081)
• [fix][client] fix negative message re-delivery twice issue (#20750)
• [fix][client] the nullValue in msgMetadata should be true by default (#22372)
• [improve][client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder (#22541)
• [improve][client] Don't print info logs for each schema loaded by client (#23206)
• [improve][client] Swallow Conscrypt ClassNotFoundException (#20371)
• [improve][client]Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover (#23129)
• [improve][client]PIP-359:Support custom message listener executor for specific subscription (#22861)

Pulsar IO and Pulsar Functions

• [fix][fn] Make python install dependencies from requirements.txt (#20174)
• [fix][io] Upgrade mssql server docker tag in DebeziumMsSqlContainer (#23318)
• [improve][fn] Add support for overriding additionalJavaRuntimeArguments with PF_additionalJavaRuntimeArguments env (#23130)
• [fix][sql][branch-3.0] Fix UUID convert failed for JSON schema (#23292)

Others

• [feat] Add scripts for updating BK RocksDB ini files (#23178)
• [fix][log] Do not print error log if tenant/namespace does not exist when calling get topic metadata (#23291)
• [fix][log] Do not print warn log when concurrently publishing and switching ledgers (#23209)
• [fix][misc] Log Conscrypt security provider initialization warnings at debug level (#23364)
• [improve][misc] Improve AES-GCM cipher performance (#23122)
• [improve][misc] Optimize TLS performance by omitting extra buffer copies (#23115)
• [improve][proxy] Reuse authentication instance in pulsar-proxy (#23113)

Tests & CI...

v3.3.1

2024-08-01

Broker

• [feat][broker] PIP-264: Add OpenTelemetry consumer metrics (#22693)
• [feat][broker] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22961)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Remove blocking calls from Subscription.getStats (#23088)
• [fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
• [fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
• [fix][broker] The topic might reference a closed ledger (#22860)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] disable loadBalancerDirectMemoryResourceWeight by default (#22821)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] maintain last active info in memory only. (#22794)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.3] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23085)
• [fix][misc] Disable JFR based telemetry collection since it's not used (#22869)
• [fix][misc] Topic name from persistence name should decode local name (#22879)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [improve] Validate range of argument before long -> int conversion (#22830)
• [improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
• [improve][broker] Add subscription prefix for internal reader (#23044)
• [improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
• [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
• [improve][broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22779)
• [improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
• [improve][broker] Reduce number of OpenTelemetry consumer attributes (#22837)
• [improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Reuse topic OpenTelemetry attributes (#22876)
• [improve][broker] Trigger offload on topic load (#22652)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] high CPU usage caused by list topics under namespace (#23049)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
• [fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted ([#22854](#22...

v3.2.4

2024-08-01

Broker

• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] The topic might reference a closed ledger (#22860)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] maintain last active info in memory only. (#22794)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.2] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23084)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [fix][misc] Topic name from persistence name should decode local name (#22879)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
• [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] Include runtime dependencies in server distribution (#22001)
• [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Trigger offload on topic load (#22652)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
• [fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted (#22854)
• [fix][client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed (#22858)
• [improve][client] improve the class GetTopicsResult (#22766)

Pulsar IO and Pulsar Functions

• [fix][fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes (#22910)
• [fix][fn] Support compression type and crypto config for all producers in Functions and Connectors (#22950)
• [improve][fn] Make producer cache bounded and expiring in Functions/Connectors (#22945)

Others

• [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
• [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
• [fix][cli] Fix expiration of tokens created with "pulsar tokens create" (#22815)
• [fix][cli] Fix the pulsar-daemon parameter passthrough syntax (#22905)
• [fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)
• [fix][offload] Break the fillbuffer loop when met EOF (#22722)

Library updates

• [fix][sec] Upgrade Bouncycastle libraries to address CVEs (#22826)
• [improve] Upgrade IPAddress to 5.5.0 (#22886)
• [improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
• [cleanup][misc] Remove classifier from netty-transport-native-unix-common dependency (#22951)
• [fix][misc] Remove RoaringBitmap dependency from pulsar-common (#23008)
• [fix][misc] Rename netty native libraries in pulsar-client-admin-shaded (#22954)
• [improve][misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin (#22957)
• [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
• [improve][misc][branch-3.2] Upgrade to Bookkeeper 4.16.6 (#22963)

Tests...

v3.0.6

2024-08-01

Broker

• [feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816) (#22740)
• [feat][broker][branch-3.0] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22960)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Fix wrong logic of method TopicName.getPartition(int index) (#19841)
• [fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Remove blocking calls from Subscription.getStats (#23088)
• [fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.0] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23077)
• [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) (#22900)
• [improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
• [improve][broker] Follow up #4196 use PulsarByteBufAllocator handle OOM (#20837)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
• [improve][broker] Improve efficiency of checking message deletion (#20490)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] Include runtime dependencies in server distribution (#22001)
• [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
• [improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] [break change] Do not create partitioned DLQ/Retry topic automatically (#22705)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] checkTopicExists supports checking partitioned topic without index (#21701)
• [improve][broker] do not grant permission for each partition to reduce unnecessary zk metadata (#18222)
• [improve][broker] high CPU usage caused by list topics under namespace (#23049)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][admin] Clearly define REST API on Open API (#22783)
• [fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
• [fix][admin] Clearly define REST API on Open API for Topics (#22782)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
• [fix][admin][part-1]Clearly define REST API on Open API (#22774)
• [improve][admin] Check if the topic existed before the permission operations (#22742)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix o...

v3.3.0

PIP

• PIP-315: Configurable max delay limit for delayed delivery (#21798)
• PIP-321 Introduce allowed-cluster at the namespace level (#21648)
• PIP-324: Alpine Docker images (#21716)
• PIP-325: Add command to abort transaction (#21731)
• PIP-326: Create a BOM to ease dependency management (#21747)
• PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
• PIP-330: getMessagesById gets all messages (#21873)
• PIP 342: Support OpenTelemetry metrics in Pulsar client (#22178)
• PIP-343: Use picocli instead of jcommander (#22181)
• PIP-344 Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
• PIP-335: Oxia metadata plugin (#22009)
• PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)

Broker

• [admin][broker] Fix force delete subscription not working (#22423)
• [cleanup][admin] Remove unused methods in PersistentTopicsBase (#22424)
• [cleanup][broker] Remove unused NamespaceBundleFactory parameter when creating OwnershipCache (#22482)
• [cleanup][broker]∂ fix doc for TransactionBuffer#isTxnAborted (#21956)
• [cleanup][broker] fix return value of TransactionTimeoutTrackerImpl#addTransaction (#22020)
• [cleanup][broker] remove useless code comment (#22459)
• [cleanup][meta] Remove com.beust.jcommander.internal import (#22294)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [cleanup][test] remove useless TestAuthorizationProvider2 (#22595)
• [feat][admin] Enable Gzip Compression by Default in Admin Client (#22464)
• [feat][admin] PIP-330: getMessagesById gets all messages (#21918)
• [feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816)
• [feat][broker] PIP-264: Add Java runtime metrics (#22616)
• [feat][broker] PIP-264: Add topic messaging metrics (#22467)
• [feat][misc] Add Pulsar BOM (Bill of Materials) (#21871)
• [feat][misc] PIP-264: Implement topic lookup metrics using OpenTelemetry (#22058)
• [feat][misc] PIP-320: Add OpenTelemetry scaffolding (#22010)
• [fix] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [fix] Fixed implicit conversions from long -> int (#22055)
• [fix] Include swagger annotations in shaded client lib (#22570)
• [fix] Restored method as deprecated in AbstractMetadataStore (#21950)
• [fix] Test was leaving client instance to null (#22631)
• [fix][admin] Clearly define REST API on Open API (#22783)
• [fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
• [fix][admin] Clearly define REST API on Open API for Topics (#22782)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix deprecated check (#22653)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][admin][part-1]Clearly define REST API on Open API (#22774)
• [fix][broker] Add the missed opentelemetry-sdk-testing dependency to tests of pulsar-broker-auth-sasl (#22665)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection (#21187)
• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Continue closing even when executor is shut down (#22599)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Delete compacted ledger when topic is deleted (#21745)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Do not migrate internal topics during Blue-Green Migration when ExtensibleLoadBalancer is used (#22478)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency (#22090)
• [fix][broker] Expire messages according to ledger close time to avoid client clock skew (#21940)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix PersistentSubscription duplicate implementation interface Subscription (#22359)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix String wrong format (#21829)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 (#21885)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled (#22139)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp (#22025)
• [fix][broker] Fix compaction/replication data loss when expire messages (#21865)
• [fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix deadlock while skip non-recoverable ledgers. (#21915)
• [fix][broker] Fix getMessageById throws 500 (#21919)
• [fix][broker] Fix hash collision when using a consumer name that ends with a number (#22053)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger (#21600)
• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured (#21894)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix potential NPE when disabling the liveness check (#21840)
• [fix][broker] Fix reader stuck when read from compacted topic with read compact mode disable (#21969)
• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set (#21842)
• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema (#21977)
• [fix][broker] Fix thread safety of loadSheddingTask and loadResourceQuotaTask fields (#22660)
• [fix][broker] Fix typo (#21843)
• [fix][broker] Fix typo in CompactedTopicImpl. (#22235)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] Fix typos in PersistentTopic class (#22364)
• [fix][broker] Fix typos in UniformLoadShedder class (#21907)
• [fix][broker] Fix write all compacted out entry into compacted topic (#21917)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Replication stopped due to unload topic failed (#21947)
• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases (#21937)
• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init (#22112)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs (#22064)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379)
• [fix][broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog (#22019)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] Support running docker container with gid != 0 (#22081)
• [fix][broker] Unify topic-level policies enable judgment conditions (#19501)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTop...