Skip to content

Commit

Permalink
Update documentation [skip ci]
Browse files Browse the repository at this point in the history
Update references to consumer groups since they are now implemented.
  • Loading branch information
tylertreat committed Mar 11, 2022
1 parent 76d4026 commit fd42439
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 11 deletions.
18 changes: 10 additions & 8 deletions documentation/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,10 @@ to the log.
> Also, a paused or starved consumer, potentially a Pod in Kubernetes, like the
> potential reporting consumer, could easily pick up where it left off when
> things slow down. Consumers may use [cursors](./cursors.md) to track their
> state, i.e. the offset. In the future, Liftbridge will support durable
> consumer groups which will allow consumers to eschew cursor management.
> state, i.e. the offset. However, the preferred pattern for tracking state is
> to use [consumer groups](./consumer_groups.md). Consumer groups can be used
> to implement [durable consumers](./consumer_groups.md#durable-consumer),
> which transparently handle cursor management.
### Scalability

Expand Down Expand Up @@ -303,18 +305,18 @@ is no bookkeeping done by the server, aside from the in-memory objects tied to
the lifecycle of the subscription. As a result, the server does not track the
position of a client in the log beyond the scope of a subscription. Instead,
Liftbridge provides a [cursors](./cursors.md) API which allows consumers to
checkpoint their position in the log and pick up where they left off. Stateful
consumer groups will be coming in the near future which will provide a more
managed solution to fault-tolerant consumption of streams.
checkpoint their position in the log and pick up where they left off. The
cursors API is used by stateful [consumer groups](./consumer_groups.md) which
provide a more managed solution to fault-tolerant consumption of streams.

> **Architect's Note**
>
> This ties back to the previously described reporting worker starved but
> clinging to an *offset* so as not to lose probable state. With _cursors_, the
> reporting worker can be restarted without state but can resume from where it
> left off due to state stored by the server via the cursors API. When stateful
> consumer groups are implemented, this will be entirely transparent to the
> consumer.
> left off due to state stored by the server via the cursors API. With
> _consumer groups_, which maintain state, this would be entirely transparent
> to the consumer.
### Stream Retention and Compaction

Expand Down
12 changes: 9 additions & 3 deletions documentation/scalability_configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,15 @@ how the scalability of these two concerns interrelate.
There are a few different ways streams can be scaled in Liftbridge. These
different approaches are discussed [here](./concepts.md#scalability). To
summarize, both stream data and consumption can be scaled horizontally by
adding additional nodes to the cluster along with additional streams or stream
partitioning. However, adding nodes to the cluster has implications with the
metadata Raft cluster used by the control plane. This is discussed below.
adding additional nodes to the cluster along with additional streams or
[stream partitioning](./concepts.md#streams-and-partitions). However, adding
nodes to the cluster has implications with the metadata Raft cluster used by
the control plane. This is discussed below.

Stream partitioning provides increased parallelism which allows for greater
throughput of messages. [Consumer groups](./consumer_groups.md) provide a means
for coordinating and balancing the consumption of partitions across a set of
consumers.

## Scaling the Control Plane

Expand Down

0 comments on commit fd42439

Please sign in to comment.