v2.8.2

This release reverts the producer changes of v2.8.1 because they has a huge impact on performance. We will reconsider the changes and make sure the performance tests accurately cover producing events before applying this change again.

What's Changed

• Revert back to 2.8.0 sendFromQueue implementation in producer by @ytalashko in #1304

Full Changelog: v2.8.1...v2.8.2

v2.8.1

Version 2.8.1 has an important bugfix that prevents partitions from being consumed twice in some circumstances (#1294). Many thanks to @ytalashko for finding the root cause, providing a fix, and tirelessly discussing the problem until it was fully understood and the fix applied.

🚀 Noteworthy changes

• Fix assign followed by fast revoke during rebalance @ytalashko (#1294)
• Add alternative fetch strategy for many partitions @erikvanoosten (#1281)
• Document metrics and consumer tuning based on metrics @erikvanoosten (#1280)

🚅 More changes

• Prevent users from enabling auto commit @erikvanoosten (#1290)
• Alternative producer implementation @erikvanoosten (#1285)
• Small improvements to the Producer @erikvanoosten (#1272)
• Improve example app @erikvanoosten (#1279)

🌱 Maintenance

• Update sbt-tpolecat to 0.5.2 @zio-scala-steward (#1301)
• Update embedded-kafka to 3.7.1.1 @zio-scala-steward (#1300)
• Update logback-classic to 1.5.7 @zio-scala-steward (#1299)
• Update zio-streams, zio-test-sbt to 2.1.7 @zio-scala-steward (#1297)
• Update zio, zio-streams, zio-test, ... to 2.1.7 @zio-scala-steward (#1295)
• Update scalafmt-core to 3.8.3 @zio-scala-steward (#1291)
• Update README.md @zio-assistant (#1278)
• Update README.md @zio-assistant (#1277)
• Update zio-kafka, zio-kafka-testkit to 2.8.0 @zio-scala-steward (#1274)
• Update zio-streams, zio-test-sbt to 2.1.6 @zio-scala-steward (#1275)

v2.8.0

Version 2.7.5 was not fully compatible with 2.7.4 which we missed during the release. Therefore, with some delay, we bump the minor version.

Moving back to pre-2.7.0 behavior: in this release, upon a lost partition the stream ends without error. To detect lost partitions, you can configure a rebalance listener.

🚀 Changes

• Allow for some auth exceptions during poll @erikvanoosten (#1270)
• A lost partition is no longer fatal @erikvanoosten (#1252)
• Update consumer docs @petern-sc (#1269)

🌱 Maintenance

• Update zio, zio-streams, zio-test, ... to 2.1.6 @zio-scala-steward (#1273)
• Update sbt to 1.10.1 @zio-scala-steward (#1271)
• Update kafka-clients to 3.7.1 @zio-scala-steward (#1268)
• Update zio-streams, zio-test-sbt to 2.1.5 @zio-scala-steward (#1267)
• Update zio, zio-streams, zio-test, ... to 2.1.5 @zio-scala-steward (#1266)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.28 @zio-scala-steward (#1263)
• Update zio-streams, zio-test-sbt to 2.1.4 @zio-scala-steward (#1261)
• Update zio, zio-streams, zio-test, ... to 2.1.4 @zio-scala-steward (#1260)
• Update scalafmt-core to 3.8.2 @zio-scala-steward (#1259)
• Update zio-streams, zio-test-sbt to 2.1.3 @zio-scala-steward (#1257)
• Update zio, zio-streams, zio-test, ... to 2.1.3 @zio-scala-steward (#1256)
• Aborts builds when a newer build is ready @erikvanoosten (#1255)
• Update README.md @zio-assistant (#1254)
• Update zio-kafka, zio-kafka-testkit to 2.7.5 @zio-scala-steward (#1253)

v2.7.5

Minor breaking changes

• breaking: Do not run user rebalance listener on same thread runtime @svroonland (#1205)
  In case you register a rebalance listener, it now runs on the default zio runtime and no longer on the special same-thread-runtime. Access to the underlying java client is removed. If this affects you, please let us know your use case.
• Remove no longer needed collection-compat @erikvanoosten (#1221)
  The library collection-compat is no longer a dependency. If you are using it, please add it to your own dependencies.

🚀 Changes

• Allow stream to be interrupted when there is no traffic @erikvanoosten (#1251)
  Lost partitions during no traffic situations are now correctly detected. That is, the stream for the partition that is lost is interrupted, the other streams are closed gracefully, the consumer aborts with an error. Note that in the next release we will change this behavior, then the stream for the partition that is lost is still ended, but the other streams will be unaffected.
• Initialize Java Producer in blocking threadpool @kyri-petrou (#1230)
• Call removeMembersFromConsumerGroup without members @sderosiaux (#1227)
• Document snapshots and Update resources @erikvanoosten (#1223)
• Drop experimental label for rebalanceSafeCommits @erikvanoosten (#1222)
• Correct prune benchmark history instructions @erikvanoosten (#1217)
• Add MIMA checks to ensure binary compatibility @erikvanoosten (#1214)

🌱 Maintenance

• Update zio-streams, zio-test-sbt to 2.1.2 @zio-scala-steward (#1249)
• Update zio, zio-streams, zio-test, ... to 2.1.2 @zio-scala-steward (#1248)
• Update zio-logging-slf4j, ... to 2.3.0 @zio-scala-steward (#1246)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.27 @zio-scala-steward (#1247)
• Update zio-logging-slf4j, ... to 2.2.4 @zio-scala-steward (#1244)
• Update zio-streams, zio-test-sbt to 2.1.1 @zio-scala-steward (#1243)
• Update zio, zio-streams, zio-test, ... to 2.1.1 @zio-scala-steward (#1242)
• Update zio-streams, zio-test-sbt to 2.1.0 @zio-scala-steward (#1241)
• Update zio, zio-streams, zio-test, ... to 2.1.0 @zio-scala-steward (#1240)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.26 @zio-scala-steward (#1237)
• Update sbt-scalafix to 0.12.1 @zio-scala-steward (#1234)
• Update sbt to 1.10.0 @zio-scala-steward (#1236)
• Update scala-library to 2.13.14 @zio-scala-steward (#1235)
• Update zio-logging-slf4j, ... to 2.2.3 @zio-scala-steward (#1226)
• Regenerate readme @erikvanoosten (#1225)
• Update logback-classic to 1.5.6 @zio-scala-steward (#1224)
• Update scala-collection-compat to 2.12.0 @zio-scala-steward (#1220)
• Revert "Bump coursier/cache-action from 6.4.5 to 6.4.6" @erikvanoosten (#1216)
• Update zio-streams, zio-test-sbt to 2.0.22 @zio-scala-steward (#1215)
• Update logback-classic to 1.5.5 @zio-scala-steward (#1211)
• Update sbt-native-packager to 1.10.0 @zio-scala-steward (#1212)
• Update zio, zio-streams, zio-test, ... to 2.0.22 @zio-scala-steward (#1213)
• Update logback-classic to 1.5.4 @zio-scala-steward (#1210)
• Update sbt-tpolecat to 0.5.1 @zio-scala-steward (#1206)
• Update scalafmt-core to 3.8.1 @zio-scala-steward (#1204)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.25 @zio-scala-steward (#1200)
• Update zio-kafka, zio-kafka-testkit to 2.7.4 @zio-scala-steward (#1197)
• Update README.md @zio-assistant (#1195)
• Bump benchmark-action/github-action-benchmark from 1.20.1 to 1.20.3 @dependabot (#1245)
• Bump coursier/cache-action from 6.4.5 to 6.4.6 @dependabot (#1209)
• Bump benchmark-action/github-action-benchmark from 1.18.0 to 1.20.1 @dependabot (#1208)

v2.7.4

Noteworthy Changes

• Update kafka-clients to 3.7.0 @zio-scala-steward (#1191)

Changes

• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.24 @zio-scala-steward (#1194)
• Update kafka-clients to 3.7.0 @zio-scala-steward (#1191)
• Update scala3-library to 3.3.3 @zio-scala-steward (#1181)
• Update sbt-scalafix to 0.12.0 @zio-scala-steward (#1184)
• Update logback-classic to 1.5.3 @zio-scala-steward (#1190)
• Update logback-classic to 1.5.2 @zio-scala-steward (#1185)
• Update logback-classic to 1.5.1 @zio-scala-steward (#1178)
• Fix a merge error and regenerate actions @erikvanoosten (#1177)
• Update README.md @zio-assistant (#1176)
• Update README.md @zio-assistant (#1175)
• Update README.md @zio-assistant (#1174)
• Update zio-streams, zio-test-sbt to 2.0.21 @zio-scala-steward (#1167)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.23 @zio-scala-steward (#1166)
• Update README.md @zio-assistant (#1173)
• Update scalafmt-core to 3.8.0 @zio-scala-steward (#1171)
• Update README.md @zio-assistant (#1172)
• Update sbt to 1.9.9 @zio-scala-steward (#1170)
• Update scala-library to 2.13.13 @zio-scala-steward (#1169)
• Update zio-logging-slf4j, ... to 2.2.2 @zio-scala-steward (#1165)
• Update logback-classic to 1.5.0 @zio-scala-steward (#1163)
• Update zio-kafka, zio-kafka-testkit to 2.7.3 @zio-scala-steward (#1164)

🌱 Dependency Updates

• Bump scala-steward-org/scala-steward-action from 2.64.0 to 2.65.0 @dependabot (#1192)
• Bump scala-steward-org/scala-steward-action from 2.63.0 to 2.64.0 @dependabot (#1186)
• Bump release-drafter/release-drafter from 5 to 6 @dependabot (#1189)
• Bump scala-steward-org/scala-steward-action from 2.61.0 to 2.63.0 @dependabot (#1133)

v2.7.3

Noteworthy Changes

• Collect consumer metrics @erikvanoosten (#1143)

Zio-kafka now passes a lot of metrics to zio. Use any zio metrics backend to expose them! See the issue #1143 for more information on which metrics are collected.

Changes

• Update zio, zio-streams, zio-test, ... to 2.0.21 @zio-scala-steward (#1144)
• Add scala steward badge (2nd attempt) @erikvanoosten (#1157)
• Update README.md @zio-assistant (#1155)
• Ensure that fibers forked in Runloop.make are interruptible @svroonland (#1154)
• Add scala steward badge @erikvanoosten (#1145)
• Collect consumer metrics @erikvanoosten (#1143)
• Update zio-logging-slf4j, ... to 2.2.0 @zio-scala-steward (#1152)
• Update zio-logging-slf4j, ... to 2.1.17 @zio-scala-steward (#1151)
• Update README.md @zio-assistant (#1150)
• Update zio-kafka, zio-kafka-testkit to 2.7.2 @zio-scala-steward (#1149)

v2.7.2

Note worthy changes

• Add partitionsFor method to producer @kyri-petrou (#1147)
• retry Consumer.CommitTimeout exception @myazinn (#1140)

Other changes

• Use Semaphore @adamgfraser (#1142)
• Make tests faster @erikvanoosten (#1139)
• Delete RunloopTimeout and small improvements @erikvanoosten (#1141)
• Update kafka-clients to 3.6.1 @zio-scala-steward (#1131)
• Improve support and documentation for manual offsets @erikvanoosten (#1135)
• Update sbt to 1.9.8 @zio-scala-steward (#1138)
• Update zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1137)
• Update zio, zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1136)
• Update logback-classic to 1.4.14 @zio-scala-steward (#1129)
• Update logback-classic to 1.3.14 @zio-scala-steward (#1128)
• Update zio-kafka, zio-kafka-testkit to 2.7.1 @zio-scala-steward (#1127)
• Remove unused config for Mergify @erikvanoosten (#1124)
• Adds test for stopping consumption while doing async commits @erikvanoosten (#1123)
• Update README.md @zio-assistant (#1125)
• Update logback-classic to 1.4.13 @zio-scala-steward (#1122)

v2.7.1

Introducing Rebalance Safe Commits

This release introduces a new experimental mode that prevent duplicate message processing during a rebalance. This mode has been in the making for 8 months and has been implemented over 9 different PRs.

We have marked Rebalance Safe Commits as an experimental feature because zio-kafka is used in many different and sometimes surprising ways. Some of those are not compatible with the new mode.

RebalanceSafeCommits can be enabled via ConsumerSettings.withRebalanceSafeCommits. Please read the scaladocs to see if this mode is applicable to how you use zio-kafka.

Much more information on the new mode (motivation, why it works, the challenges, etc.), are available in the descriptions of PR Await commits during a rebalance. A manual is forthcoming.

Changes

• Make maxRebalanceDuration configurable @erikvanoosten (#1118)
• Update zio-logging-slf4j, ... to 2.1.16 @zio-scala-steward (#1114)
• Automatically add new flame graphs to gh_pages, prune old @erikvanoosten (#1105)
• Await commit during a rebalance @erikvanoosten (#1098)
• Update README.md @zio-assistant (#1113)
• Update zio-kafka, zio-kafka-testkit to 2.7.0 @zio-scala-steward (#1112)

v2.7.0

Improvements

Zio-kafka became a little bit faster by doing less work when there is no rebalance.

Really large and tiny messages are better described in the tuning documentation, and ConsumerSettings.withoutPartitionPreFetching was introduced to make it easier to disable pre-fetching.

Previously, in some use cases, you could end up with a live-lock trying to get access to the underlying java consumer. This should no longer happen.

Breaking changes

Zio-kafka no longer supports scala 2.12. Scala 2.13 has been available for 4½ year so everyone has had plenty of time to move to a more modern scala version. (#1104)

This release has some some minor breaking changes:

• The diagnostics events for rebalances have been improved in an incompatible way (#1102)
• Jackson was removed as a dependency (#1096)
• partitionPreFetchBufferLimit is now 0-based instead of 1-based (#1091)

Changes

• Update scalafmt-core to 3.7.17 @zio-scala-steward (#1111)
• Make access to the java consumer fair @flavienbert (#1109)
• Extend tuning docs @erikvanoosten (#1110)
• Protect agains user diagnostics, better rebalance events @erikvanoosten (#1102)
• Drop support for scala 2.12 @erikvanoosten (#1104)
• Update zio-streams, zio-test-sbt to 2.0.19 @zio-scala-steward (#1101)
• Update zio, zio-streams, zio-test, ... to 2.0.19 @zio-scala-steward (#1100)
• Update zio-logging-slf4j, ... to 2.1.15 @zio-scala-steward (#1099)
• Track latest completed commit offset per partition @erikvanoosten (#1097)
• Always end streams in rebalance listener, support lost partitions @erikvanoosten (#1089)
• Remove jackson dependency @erikvanoosten (#1096)
• Make partitionPreFetchBufferLimit value 0 disable prefetching @erikvanoosten (#1091)
• Update zio-kafka, zio-kafka-testkit to 2.6.0 @zio-scala-steward (#1092)
• Update README.md @zio-assistant (#1090)
• Prevent double logging in tests @erikvanoosten (#1088)

v2.6.0

Noticeable Changes

• Automatically merge commits by @erikvanoosten in #1073
  In this PR zio-kafka automatically merges all commits that were generated in the course of a single run of the runloop. This frees users from having to merge streams and do the commit merging themselves.
• Add processing timeout at the stream level by @erikvanoosten in #1009
  Say good bye to the runloop-timeout! From this version on we respect Kafka configuration max.poll.interval.ms. When a stream does not consume any records for max.poll.interval.ms or longer (while records are available), the stream will be interrupted and the consumer closed. This allows another member of the consumer group to take over the work.
• Add tuning options and docs by @erikvanoosten in #1081
  A new tuning guide on https://zio.dev/zio-kafka/.
• Enable the consumer to commit an offset with metadata by @flavienbert in #1067

See the PR descriptions for more information.

All Changes

• Update zio-kafka, zio-kafka-testkit to 2.5.0 by @zio-scala-steward in #1070
• Update README.md by @zio-assistant in #1071
• Gitignore cleanup by @erikvanoosten in #1072
• Small cleanup by @erikvanoosten in #1065
• Do not run CI and benches for gh-pages branch by @erikvanoosten in #1075
• Add instructions for pruning the benchmark history by @erikvanoosten in #1077
• Automatically merge commits by @erikvanoosten in #1073
• Update zio-sbt to 0.4.0-alpha.22 by @erikvanoosten in #1078
• Update jackson-databind to 2.15.3 by @zio-scala-steward in #1080
• Add processing timeout at the stream level by @erikvanoosten in #1009
• Fix flaky test by @erikvanoosten in #1082
• Fix flaky test again by @erikvanoosten in #1083
• Add tuning options and docs by @erikvanoosten in #1081
• Add unit test for QueueSizeBasedFetchStrategy by @erikvanoosten in #1017
• Update sbt to 1.9.7 by @zio-scala-steward in #1084
• Update scalafmt-core to 3.7.15 by @zio-scala-steward in #1085
• Enable the consumer to commit an offset with metadata by @flavienbert in #1067
• Keep track of last pulled offset by @erikvanoosten in #1086

New Contributors

• @flavienbert made their first contribution in #1067

Full Changelog: v2.5.0...v2.6.0