Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kinesis: Native provider #1601

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
+33 −14
Draft

Kinesis: Native provider #1601

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
20e8308
Kinesis: Add new provider
viren-nadkarni Dec 17, 2024
6ad2eb3
Rename provider from 'nextgen' to 'native'
viren-nadkarni Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 33 additions & 14 deletions content/en/user-guide/aws/kinesis/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,22 @@
---
title: "Kinesis"
linkTitle: "Kinesis"
description: Get started with Kinesis on LocalStack
title: "Kinesis Data Streams"
linkTitle: "Kinesis Data Streams"
description: Get started with Kinesis Data Streams on LocalStack
persistence: supported

---

## Introduction

Kinesis is a platform provided by Amazon Web Services (AWS) that enables your application to ingest, buffer, and process data in real-time.
Kinesis is suitable for applications that require processing and deriving insights from data streams such as logs, metrics, user interactions, and IoT sensor readings.
Kinesis offers three main services: Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics.
In this page, we take a look at Kinesis Data Streams which allows you to capture and store real-time data streams.
Kinesis Data Streams is an AWS service for ingesting, buffering, and processing data in high throughput data streams.
It is used for applications that require real-time processing and deriving insights from data streams such as logs, metrics, user interactions, and sensor readings.

LocalStack allows you to use the Kinesis APIs in your local environment from setting up data streams and configuring data processing to building real-time analytics applications.
The supported APIs are available on our [API coverage page](https://docs.localstack.cloud/references/coverage/coverage_kinesis/), which provides information on the extent of Kinesis's integration with LocalStack.
LocalStack allows you to use the Kinesis Data Streams APIs in your local environment from setting up data streams and configuring data processing to building real-time applications.
The supported APIs are available on our [API coverage page](https://docs.localstack.cloud/references/coverage/coverage_kinesis/).

## Getting started

This guide is designed for users new to Kinesis and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.
This guide is designed for users new to Kinesis Data Streams and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.

Start your LocalStack container using your preferred method.
We will demonstrate how to create a Lambda function to consume events from a Kinesis stream with the AWS CLI.
Expand Down Expand Up @@ -171,7 +169,16 @@

You can fetch the CloudWatch logs for your Lambda function reading records from the stream, using AWS CLI or LocalStack Resource Browser.

## Configuration
## Implementations

There are two implementations of Kinesis Data Streams available in LocalStack.

### Kinesis Mock

This is the default provider for Kinesis Data Streams on LocalStack.
It is uses [Kinesis Mock](https://github.com/etspaceman/kinesis-mock) under the hood.

This provider has the following configuration options:

| Variable | Description |
| -------- | ----------- |
Expand All @@ -180,6 +187,11 @@
| `KINESIS_LATENCY` | Integer value in milliseconds (default: `500`) or `0` (to disable). Particularly useful for testing latency-sensitive applications. Since local Kinesis service lacks latency simulation, you can introduce artificial latency into your AWS calls using this variable. To disable this behavior, set `KINESIS_LATENCY=0`. |
| `KINESIS_INITIALIZE_STREAMS` | **Deprecated.** Comma-delimited string with stream names, corresponding shard counts, and an optional region for initialization during startup. If no region is provided, the default region is used. For example, `KINESIS_INITIALIZE_STREAMS=my-first-stream:1,my-other-stream:2:us-west-2,my-last-stream:1`. In multi-account setups, the specified streams will be created for all accounts. |

### Native

This provider is implemented purely in Python.
It can be enabled by setting the configuration option `PROVIDER_OVERRIDE_KINESIS=native`

## Resource Browser

The LocalStack Web Application provides a Resource Browser for managing Kinesis Streams & Kafka Clusters.
Expand All @@ -204,7 +216,14 @@
- [Search application with Lambda, Kinesis, Firehose, ElasticSearch, S3](https://github.com/localstack/sample-fuzzy-movie-search-lambda-kinesis-elasticsearch)
- [Streaming Data Pipeline with Kinesis, Tinybird, CloudWatch, Lambda](https://github.com/localstack/serverless-streaming-data-pipeline)

## Current Limitations
## Limitations

### Kinesis Mock

- In multi-account setups, each AWS account launches a separate instance of Kinesis Mock, which is very resource intensive when a large number of AWS accounts are used.
An [open Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature.

### Native

- TODO

In multi-account setups, each AWS account launches a separate instance of Kinesis Mock, which is very resource intensive when a large number of AWS accounts are used.
An [open Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature.
Loading