Skip to content

Commit

Permalink
Kinesis: Add new provider
Browse files Browse the repository at this point in the history
  • Loading branch information
viren-nadkarni committed Dec 17, 2024
1 parent 82e4896 commit 20e8308
Showing 1 changed file with 33 additions and 14 deletions.
47 changes: 33 additions & 14 deletions content/en/user-guide/aws/kinesis/index.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,22 @@
---
title: "Kinesis"
linkTitle: "Kinesis"
description: Get started with Kinesis on LocalStack
title: "Kinesis Data Streams"
linkTitle: "Kinesis Data Streams"
description: Get started with Kinesis Data Streams on LocalStack
persistence: supported

---

## Introduction

Kinesis is a platform provided by Amazon Web Services (AWS) that enables your application to ingest, buffer, and process data in real-time.
Kinesis is suitable for applications that require processing and deriving insights from data streams such as logs, metrics, user interactions, and IoT sensor readings.
Kinesis offers three main services: Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics.
In this page, we take a look at Kinesis Data Streams which allows you to capture and store real-time data streams.
Kinesis Data Streams is an AWS service for ingesting, buffering, and processing data in high throughput data streams.
It is used for applications that require real-time processing and deriving insights from data streams such as logs, metrics, user interactions, and sensor readings.

LocalStack allows you to use the Kinesis APIs in your local environment from setting up data streams and configuring data processing to building real-time analytics applications.
The supported APIs are available on our [API coverage page](https://docs.localstack.cloud/references/coverage/coverage_kinesis/), which provides information on the extent of Kinesis's integration with LocalStack.
LocalStack allows you to use the Kinesis Data Streams APIs in your local environment from setting up data streams and configuring data processing to building real-time applications.
The supported APIs are available on our [API coverage page](https://docs.localstack.cloud/references/coverage/coverage_kinesis/).

## Getting started

This guide is designed for users new to Kinesis and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.
This guide is designed for users new to Kinesis Data Streams and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.

Start your LocalStack container using your preferred method.
We will demonstrate how to create a Lambda function to consume events from a Kinesis stream with the AWS CLI.
Expand Down Expand Up @@ -171,7 +169,16 @@ $ awslocal kinesis put-record \

You can fetch the CloudWatch logs for your Lambda function reading records from the stream, using AWS CLI or LocalStack Resource Browser.

## Configuration
## Implementations

There are two implementations of Kinesis Data Streams available in LocalStack.

### Kinesis Mock

This is the default provider for Kinesis Data Streams on LocalStack.
It is uses [Kinesis Mock](https://github.com/etspaceman/kinesis-mock) under the hood.

This provider has the following configuration options:

| Variable | Description |
| -------- | ----------- |
Expand All @@ -180,6 +187,11 @@ You can fetch the CloudWatch logs for your Lambda function reading records from
| `KINESIS_LATENCY` | Integer value in milliseconds (default: `500`) or `0` (to disable). Particularly useful for testing latency-sensitive applications. Since local Kinesis service lacks latency simulation, you can introduce artificial latency into your AWS calls using this variable. To disable this behavior, set `KINESIS_LATENCY=0`. |
| `KINESIS_INITIALIZE_STREAMS` | **Deprecated.** Comma-delimited string with stream names, corresponding shard counts, and an optional region for initialization during startup. If no region is provided, the default region is used. For example, `KINESIS_INITIALIZE_STREAMS=my-first-stream:1,my-other-stream:2:us-west-2,my-last-stream:1`. In multi-account setups, the specified streams will be created for all accounts. |

### Native

This provider is implemented purely in Python.
It can be enabled by setting the configuration option `PROVIDER_OVERRIDE_KINESIS=nextgen`

## Resource Browser

The LocalStack Web Application provides a Resource Browser for managing Kinesis Streams & Kafka Clusters.
Expand All @@ -204,7 +216,14 @@ The following code snippets and sample applications provide practical examples o
- [Search application with Lambda, Kinesis, Firehose, ElasticSearch, S3](https://github.com/localstack/sample-fuzzy-movie-search-lambda-kinesis-elasticsearch)
- [Streaming Data Pipeline with Kinesis, Tinybird, CloudWatch, Lambda](https://github.com/localstack/serverless-streaming-data-pipeline)

## Current Limitations
## Limitations

### Kinesis Mock

- In multi-account setups, each AWS account launches a separate instance of Kinesis Mock, which is very resource intensive when a large number of AWS accounts are used.
An [open Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature.

### Native

- TODO

In multi-account setups, each AWS account launches a separate instance of Kinesis Mock, which is very resource intensive when a large number of AWS accounts are used.
An [open Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature.

0 comments on commit 20e8308

Please sign in to comment.