From fd42439487ffcb82ff95e04b2d48e964505dd8fe Mon Sep 17 00:00:00 2001 From: Tyler Treat Date: Fri, 11 Mar 2022 13:28:22 -0700 Subject: [PATCH] Update documentation [skip ci] Update references to consumer groups since they are now implemented. --- documentation/concepts.md | 18 ++++++++++-------- documentation/scalability_configuration.md | 12 +++++++++--- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/documentation/concepts.md b/documentation/concepts.md index 36daac6a..15c74e43 100644 --- a/documentation/concepts.md +++ b/documentation/concepts.md @@ -124,8 +124,10 @@ to the log. > Also, a paused or starved consumer, potentially a Pod in Kubernetes, like the > potential reporting consumer, could easily pick up where it left off when > things slow down. Consumers may use [cursors](./cursors.md) to track their -> state, i.e. the offset. In the future, Liftbridge will support durable -> consumer groups which will allow consumers to eschew cursor management. +> state, i.e. the offset. However, the preferred pattern for tracking state is +> to use [consumer groups](./consumer_groups.md). Consumer groups can be used +> to implement [durable consumers](./consumer_groups.md#durable-consumer), +> which transparently handle cursor management. ### Scalability @@ -303,18 +305,18 @@ is no bookkeeping done by the server, aside from the in-memory objects tied to the lifecycle of the subscription. As a result, the server does not track the position of a client in the log beyond the scope of a subscription. Instead, Liftbridge provides a [cursors](./cursors.md) API which allows consumers to -checkpoint their position in the log and pick up where they left off. Stateful -consumer groups will be coming in the near future which will provide a more -managed solution to fault-tolerant consumption of streams. +checkpoint their position in the log and pick up where they left off. The +cursors API is used by stateful [consumer groups](./consumer_groups.md) which +provide a more managed solution to fault-tolerant consumption of streams. > **Architect's Note** > > This ties back to the previously described reporting worker starved but > clinging to an *offset* so as not to lose probable state. With _cursors_, the > reporting worker can be restarted without state but can resume from where it -> left off due to state stored by the server via the cursors API. When stateful -> consumer groups are implemented, this will be entirely transparent to the -> consumer. +> left off due to state stored by the server via the cursors API. With +> _consumer groups_, which maintain state, this would be entirely transparent +> to the consumer. ### Stream Retention and Compaction diff --git a/documentation/scalability_configuration.md b/documentation/scalability_configuration.md index 863c0bb8..02d43a7a 100644 --- a/documentation/scalability_configuration.md +++ b/documentation/scalability_configuration.md @@ -18,9 +18,15 @@ how the scalability of these two concerns interrelate. There are a few different ways streams can be scaled in Liftbridge. These different approaches are discussed [here](./concepts.md#scalability). To summarize, both stream data and consumption can be scaled horizontally by -adding additional nodes to the cluster along with additional streams or stream -partitioning. However, adding nodes to the cluster has implications with the -metadata Raft cluster used by the control plane. This is discussed below. +adding additional nodes to the cluster along with additional streams or +[stream partitioning](./concepts.md#streams-and-partitions). However, adding +nodes to the cluster has implications with the metadata Raft cluster used by +the control plane. This is discussed below. + +Stream partitioning provides increased parallelism which allows for greater +throughput of messages. [Consumer groups](./consumer_groups.md) provide a means +for coordinating and balancing the consumption of partitions across a set of +consumers. ## Scaling the Control Plane