Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update bedrock v4 docs #1554

Merged
merged 1 commit into from
Nov 18, 2024
Merged

Update bedrock v4 docs #1554

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion content/en/references/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ This section covers configuration options that are specific to certain AWS servi

| Variable | Example Values | Description |
| - | - | - |
| `LOCALSTACK_ENABLE_BEDROCK` | `1` | Use the Bedrock provider |
| `BEDROCK_PREWARM` | `0` (default) \| `1` | Pre-warm the Bedrock engine directly on LocalStack startup instead of on demand. |
| `DEFAULT_BEDROCK_MODEL` | `qwen2.5:0.5b` (default) | The model to use to handle text model invocations in Bedrock. Any text-based model available for Ollama is usable. |

### BigData (EMR, Athena, Glue)

Expand Down
19 changes: 15 additions & 4 deletions content/en/user-guide/aws/bedrock/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,27 @@ The supported APIs are available on our [API Coverage Page](https://docs.localst

This guide is designed for users new to AWS Bedrock and assumes basic knowledge of the AWS CLI and our `awslocal` wrapper script.

Start your LocalStack container using your preferred method using the `LOCALSTACK_ENABLE_BEDROCK=1` configuration variable.
Start your LocalStack container using your preferred method with or without pre-warming the Bedrock engine.
We will demonstrate how to use Bedrock by following these steps:

1. Listing available foundation models
2. Invoking a model for inference
3. Using the conversation API

### Pre-warming the Bedrock engine

The startup of the Bedrock engine can take some time.
Per default, we only start it once you send a request to one of the `bedrock-runtime` APIs.
However, if you want to start the engine when localstack starts to avoid long wait times on your first request you can set the flag `BEDROCK_PREWARM`.

### List available foundation models

You can view all available foundation models using the [`ListFoundationModels`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListFoundationModels.html) API.
This will show you which models are available for use in your local environment.
This will show you which models are available on AWS Bedrock.
{{< callout "note">}}
The actual model that will be used for emulation will differ from the ones defined in this list.
You can define the used model with `DEFAULT_BEDROCK_MODEL`
{{< / callout >}}

Run the following command:

Expand All @@ -36,7 +46,8 @@ $ awslocal bedrock list-foundation-models
### Invoke a model

You can use the [`InvokeModel`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) API to send requests to a specific model.
In this example, we'll use the Llama 3 model to process a simple prompt.
In this example, we selected the Llama 3 model to process a simple prompt.
However, the actual model will be defined by the `DEFAULT_BEDROCK_MODEL` environment variable.

Run the following command:

Expand Down Expand Up @@ -75,5 +86,5 @@ $ awslocal bedrock-runtime converse \

## Limitations

* LocalStack Bedrock implementation is mock-only and does not run any LLM model locally.
* LocalStack Bedrock currently only officially supports text-based models.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "currently only officially" mean exactly?

Copy link
Member Author

@silv-io silv-io Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That at this point in time we only officially support text-based models as opposed to image or other kinds of binary data models.

Do you have an alternative way to read this sentence? I can clarify based on that :D

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question would be what kind of models are unofficially supported 😅 . It's fine for me now, this was just the first question popping in my head. For me they are either supported or not, but this phrasing leads me to believe other models might be unofficially supported :P

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unofficially we support any model that can run in ollama, but not every single one is tested. The invoke-model endpoint accepts binary data, so you could theoretically run non text-based models and use it without any issue probably - we just haven't tested that yet :)

* Currently, GPU models are not supported by the LocalStack Bedrock implementation.