Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate upstream changes #23

Merged
merged 28 commits into from
Oct 18, 2023
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
2aea6aa
docs: removing trailing whitespace (#76)
Jan 4, 2023
de2c850
docs: adding Go version/release badges (#77)
Jan 5, 2023
4c56a13
docs: adding content table (#78)
Jan 5, 2023
ee52c16
docs: Display environment variable as code snippet
yeongrokgim Jan 5, 2023
6c827ea
docs: corrected bullet point's content spacing/indentation (#81)
Jan 5, 2023
559b115
docs: Add syntax highlighting, fix typos, other cleanups (#82)
jordan-brough Jan 17, 2023
a08886c
chore(deps): bump golang.org/x/net from 0.1.0 to 0.7.0 (#85)
dependabot[bot] Mar 3, 2023
c1cf0c5
feat: Pull upstream changes 2023/04 (#87)
valerena Apr 27, 2023
bf7e248
chore(deps): Update to Golang 1.19 (#95)
valerena Jun 8, 2023
861ab2f
Merge remote-tracking branch 'upstream/develop' into integrate-upstre…
joe4dev Oct 6, 2023
8b003ee
Remove unknown field CorrelationID for Invoke
joe4dev Oct 6, 2023
c6ac472
Remove unknown field CorrelationID for Init
joe4dev Oct 6, 2023
49a6bb0
Add new sandbox init parameters
joe4dev Oct 6, 2023
75f706b
Add bootstrap to InitHandler parameters
joe4dev Oct 6, 2023
a413742
Add new runtimeDomainRoot to NewBootstrapSingleCmd
joe4dev Oct 6, 2023
0d0a067
Fix new interop server interface (WIP)
joe4dev Oct 6, 2023
35e77fb
Remove SetTailLogOutput
joe4dev Oct 6, 2023
c4e9547
Use new way of getting the default interop server
joe4dev Oct 6, 2023
691651c
Bump Golang Docker image for delve build
joe4dev Oct 6, 2023
1e9ef9d
Clarify ARM debugging limitations
joe4dev Oct 11, 2023
80f6683
Fix sanbox creation nil pointer with new interop initialization
joe4dev Oct 11, 2023
ed9ea3e
Add status ready to LocalStack after await initialized (WIP)
joe4dev Oct 13, 2023
6b9d9a6
Add custom LocalStack LogsEgressAPI using logCollector
joe4dev Oct 13, 2023
9fabdc9
Improve logging for runtime init initialization
joe4dev Oct 16, 2023
b273ca6
Merge branch 'localstack' into integrate-upstream-changes
joe4dev Oct 16, 2023
95ee607
Fix missing request metadata in reserve call
joe4dev Oct 17, 2023
cee0c69
Add changelog entry for awsutil
joe4dev Oct 17, 2023
37c4634
Fix init error payload reporting
joe4dev Oct 17, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ compile-lambda-linux-all:
make ARCH=arm64 compile-lambda-linux

compile-with-docker:
docker run --rm --env GOPROXY=direct -v $(shell pwd):/LambdaRuntimeLocal -w /LambdaRuntimeLocal golang:1.18 make ARCH=${ARCH} compile-lambda-linux
docker run --rm --env GOPROXY=direct -v $(shell pwd):/LambdaRuntimeLocal -w /LambdaRuntimeLocal golang:1.19 make ARCH=${ARCH} compile-lambda-linux

compile-lambda-linux:
CGO_ENABLED=0 GOOS=linux GOARCH=${GO_ARCH_${ARCH}} go build -ldflags "${RELEASE_BUILD_LINKER_FLAGS}" -gcflags="${GC_FLAGS}" -o ${DESTINATION_${ARCH}} ./cmd/localstack
Expand Down
104 changes: 65 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
## AWS Lambda Runtime Interface Emulator
![GitHub release (latest by date)](https://img.shields.io/github/v/release/aws/aws-lambda-runtime-interface-emulator)
![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/aws/aws-lambda-runtime-interface-emulator)
![GitHub](https://img.shields.io/github/license/aws/aws-lambda-runtime-interface-emulator)

![Apache-2.0](https://img.shields.io/npm/l/aws-sam-local.svg)

The Lambda Runtime Interface Emulator is a proxy for Lambda’s Runtime and Extensions APIs, which allows customers to
locally test their Lambda function packaged as a container image. It is a lightweight web-server that converts
Expand All @@ -12,6 +14,21 @@ requests instead of the JSON events required for deployment to Lambda. This comp
Lambda’s orchestrator, or security and authentication configurations. You can get started by downloading and installing it on your local machine. When the Lambda Runtime API emulator is executed, a `/2015-03-31/functions/function/invocations` endpoint will be stood up within the container that you post data to it in order to invoke your function for testing.


## Content
* [Installing](#installing)
* [Getting started](#getting-started)
* [Test an image with RIE included in the image](#test-an-image-with-rie-included-in-the-image)
* [To test your Lambda function with the emulator](#to-test-your-lambda-function-with-the-emulator)
* [Build RIE into your base image](#build-rie-into-your-base-image)
* [To build the emulator into your image](#to-build-the-emulator-into-your-image)
* [Test an image without adding RIE to the image](#test-an-image-without-adding-rie-to-the-image)
* [To test an image without adding RIE to the image](#to-test-an-image-without-adding-rie-to-the-image)
* [How to configure](#how-to-configure)
* [Level of support](#level-of-support)
* [Security](#security)
* [License](#license)


## Installing

Instructions for installing AWS Lambda Runtime Interface Emulator for your platform
Expand All @@ -26,26 +43,26 @@ Instructions for installing AWS Lambda Runtime Interface Emulator for your platf

## Getting started

There are a few ways you use the Runtime Interface Emulator (RIE) to locally test your function depending on the base image used.
There are a few ways you use the Runtime Interface Emulator (RIE) to locally test your function depending on the base image used.


### Test an image with RIE included in the image

The AWS base images for Lambda include the runtime interface emulator. You can also follow these steps if you built the RIE into your alternative base image.
The AWS base images for Lambda include the runtime interface emulator. You can also follow these steps if you built the RIE into your alternative base image.

#### To test your Lambda function with the emulator

1. Build your image locally using the docker build command.
1. Build your image locally using the docker build command.

`docker build -t myfunction:latest .`

2. Run your container image locally using the docker run command.
2. Run your container image locally using the docker run command.

`docker run -p 9000:8080 myfunction:latest`

This command runs the image as a container and starts up an endpoint locally at `localhost:9000/2015-03-31/functions/function/invocations`.
This command runs the image as a container and starts up an endpoint locally at `localhost:9000/2015-03-31/functions/function/invocations`.

3. Post an event to the following endpoint using a curl command:
3. Post an event to the following endpoint using a curl command:

`curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'`

Expand All @@ -59,10 +76,11 @@ You can build RIE into a base image. Download the RIE from GitHub to your local

1. Create a script and save it in your project directory. Set execution permissions for the script file.

The script checks for the presence of the `AWS_LAMBDA_RUNTIME_API` environment variable, which indicates the presence of the runtime API. If the runtime API is present, the script runs [the runtime interface client](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-images.html#runtimes-api-client). Otherwise, the script runs the runtime interface emulator.
The script checks for the presence of the `AWS_LAMBDA_RUNTIME_API` environment variable, which indicates the presence of the runtime API. If the runtime API is present, the script runs [the runtime interface client](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-images.html#runtimes-api-client). Otherwise, the script runs the runtime interface emulator.

The following example shows a typical script for a Node.js function.
```
The following example shows a typical script for a Node.js function.

```sh
#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
exec /usr/local/bin/aws-lambda-rie /usr/bin/npx aws-lambda-ric
Expand All @@ -71,74 +89,83 @@ The following example shows a typical script for a Node.js function.
fi
```

2. Download the [runtime interface emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest) for your target architecture (`aws-lambda-rie` for x86\_64 or `aws-lambda-rie-arm64` for arm64) from GitHub into your project directory.
2. Download the [runtime interface emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest) for your target architecture (`aws-lambda-rie` for x86\_64 or `aws-lambda-rie-arm64` for arm64) from GitHub into your project directory.

3. Install the emulator package and change `ENTRYPOINT` to run the new script by adding the following lines to your Dockerfile:

To use the default x86\_64 architecture
```
To use the default x86\_64 architecture

```dockerfile
ADD aws-lambda-rie /usr/local/bin/aws-lambda-rie
ENTRYPOINT [ "/entry_script.sh" ]
```

To use the arm64 architecture:
```
To use the arm64 architecture:

```dockerfile
ADD aws-lambda-rie-arm64 /usr/local/bin/aws-lambda-rie
ENTRYPOINT [ "/entry_script.sh" ]
```

4. Build your image locally using the docker build command.
```

```sh
docker build -t myfunction:latest .
```

5. Run your image locally using the docker run command.
```

```sh
docker run -p 9000:8080 myfunction:latest
```

### Test an image without adding RIE to the image

You install the runtime interface emulator to your local machine. When you run the container image, you set the entry point to be the emulator.
*To test an image without adding RIE to the image *

#### To test an image without adding RIE to the image

1. From your project directory, run the following command to download the RIE (x86-64 architecture) from GitHub and install it on your local machine.

```
```sh
mkdir -p ~/.aws-lambda-rie && curl -Lo ~/.aws-lambda-rie/aws-lambda-rie \
https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie \
&& chmod +x ~/.aws-lambda-rie/aws-lambda-rie
```
```

To download the RIE for arm64 architecture, use the previous command with a different GitHub download url.

To download the RIE for arm64 architecture, use the previous command with a different GitHub download url.
```
https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie-arm64 \
```

2. Run your Lambda image function using the docker run command.
```
docker run -d -v ~/.aws-lambda-rie:/aws-lambda -p 9000:8080 myfunction:latest
--entrypoint /aws-lambda/aws-lambda-rie <image entrypoint> <(optional) image command>`
2. Run your Lambda image function using the docker run command.

```sh
docker run -d -v ~/.aws-lambda-rie:/aws-lambda -p 9000:8080 myfunction:latest \
--entrypoint /aws-lambda/aws-lambda-rie <image entrypoint> <(optional) image command>
```

This runs the image as a container and starts up an endpoint locally at `localhost:9000/2015-03-31/functions/function/invocations`.
This runs the image as a container and starts up an endpoint locally at `localhost:9000/2015-03-31/functions/function/invocations`.

3. Post an event to the following endpoint using a curl command:
3. Post an event to the following endpoint using a curl command:

`curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'`
```sh
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
```

This command invokes the function running in the container image and returns a response.

## How to configure
## How to configure

`aws-lambda-rie` can be configured through Environment Variables within the local running Image.
`aws-lambda-rie` can be configured through Environment Variables within the local running Image.
You can configure your credentials by setting:
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_SESSION_TOKEN`
* `AWS_REGION`

You can configure timeout by setting AWS_LAMBDA_FUNCTION_TIMEOUT to the number of seconds you want your function to timeout in.
You can configure timeout by setting `AWS_LAMBDA_FUNCTION_TIMEOUT` to the number of seconds you want your function to timeout in.

The rest of these Environment Variables can be set to match AWS Lambda's environment but are not required.
* `AWS_LAMBDA_FUNCTION_VERSION`
Expand All @@ -147,17 +174,16 @@ The rest of these Environment Variables can be set to match AWS Lambda's environ

## Level of support

You can use the emulator to test if your function code is compatible with the Lambda environment, executes successfully
and provides the expected output. For example, you can mock test events from different event sources. You can also use
it to test extensions and agents built into the container image against the Lambda Extensions API. This component
does *not *emulate* *the orchestration behavior of AWS Lambda. For example, Lambda has a network and security
configurations that will not be emulated by this component.

You can use the emulator to test if your function code is compatible with the Lambda environment, executes successfully
and provides the expected output. For example, you can mock test events from different event sources. You can also use
it to test extensions and agents built into the container image against the Lambda Extensions API. This component
does _not_ emulate the orchestration behavior of AWS Lambda. For example, Lambda has a network and security
configurations that will not be emulated by this component.

* You can use the emulator to test if your function code is compatible with the Lambda environment, runs successfully and provides the expected output.
* You can also use it to test extensions and agents built into the container image against the Lambda Extensions API.
* This component does _not_ emulate Lambda’s orchestration, or security and authentication configurations.
* The component does _not_ support X-ray and other Lambda integrations locally.
* This component does _not_ emulate Lambda’s orchestration, or security and authentication configurations.
* The component does _not_ support X-ray and other Lambda integrations locally.
* The component supports only Linux, for x86-64 and arm64 architectures.

## Security
Expand Down
31 changes: 25 additions & 6 deletions cmd/aws-lambda-rie/handlers.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ import (
"strings"
"time"

"go.amzn.com/lambda/core/statejson"
"go.amzn.com/lambda/interop"
"go.amzn.com/lambda/rapidcore"
"go.amzn.com/lambda/rapidcore/env"

"github.com/google/uuid"

Expand All @@ -27,6 +29,19 @@ type Sandbox interface {
Invoke(responseWriter http.ResponseWriter, invoke *interop.Invoke) error
}

type InteropServer interface {
Init(i *interop.Init, invokeTimeoutMs int64) error
AwaitInitialized() error
FastInvoke(w http.ResponseWriter, i *interop.Invoke, direct bool) error
Reserve(id string, traceID, lambdaSegmentID string) (*rapidcore.ReserveResponse, error)
Reset(reason string, timeoutMs int64) (*statejson.ResetDescription, error)
AwaitRelease() (*statejson.InternalStateDescription, error)
Shutdown(shutdown *interop.Shutdown) *statejson.InternalStateDescription
InternalState() (*statejson.InternalStateDescription, error)
CurrentToken() *interop.Token
Restore(restore *interop.Restore) error
}

var initDone bool

func GetenvWithDefault(key string, defaultValue string) string {
Expand Down Expand Up @@ -57,7 +72,7 @@ func printEndReports(invokeId string, initDuration string, memorySize string, in
invokeId, invokeDuration, math.Ceil(invokeDuration), memorySize, memorySize)
}

func InvokeHandler(w http.ResponseWriter, r *http.Request, sandbox Sandbox) {
func InvokeHandler(w http.ResponseWriter, r *http.Request, sandbox Sandbox, bs interop.Bootstrap) {
log.Debugf("invoke: -> %s %s %v", r.Method, r.URL, r.Header)
bodyBytes, err := ioutil.ReadAll(r.Body)
if err != nil {
Expand All @@ -80,7 +95,7 @@ func InvokeHandler(w http.ResponseWriter, r *http.Request, sandbox Sandbox) {

if !initDone {

initStart, initEnd := InitHandler(sandbox, functionVersion, timeout)
initStart, initEnd := InitHandler(sandbox, functionVersion, timeout, bs)

// Calculate InitDuration
initTimeMS := math.Min(float64(initEnd.Sub(initStart).Nanoseconds()),
Expand All @@ -99,7 +114,6 @@ func InvokeHandler(w http.ResponseWriter, r *http.Request, sandbox Sandbox) {
TraceID: r.Header.Get("X-Amzn-Trace-Id"),
LambdaSegmentID: r.Header.Get("X-Amzn-Segment-Id"),
Payload: bytes.NewReader(bodyBytes),
CorrelationID: "invokeCorrelationID",
}
fmt.Println("START RequestId: " + invokePayload.ID + " Version: " + functionVersion)

Expand Down Expand Up @@ -166,7 +180,7 @@ func InvokeHandler(w http.ResponseWriter, r *http.Request, sandbox Sandbox) {
w.Write(invokeResp.Body)
}

func InitHandler(sandbox Sandbox, functionVersion string, timeout int64) (time.Time, time.Time) {
func InitHandler(sandbox Sandbox, functionVersion string, timeout int64, bs interop.Bootstrap) (time.Time, time.Time) {
additionalFunctionEnvironmentVariables := map[string]string{}

// Add default Env Vars if they were not defined. This is a required otherwise 1p Python2.7, Python3.6, and
Expand All @@ -189,15 +203,20 @@ func InitHandler(sandbox Sandbox, functionVersion string, timeout int64) (time.T
// pass to rapid
sandbox.Init(&interop.Init{
Handler: GetenvWithDefault("AWS_LAMBDA_FUNCTION_HANDLER", os.Getenv("_HANDLER")),
CorrelationID: "initCorrelationID",
AwsKey: os.Getenv("AWS_ACCESS_KEY_ID"),
AwsSecret: os.Getenv("AWS_SECRET_ACCESS_KEY"),
AwsSession: os.Getenv("AWS_SESSION_TOKEN"),
XRayDaemonAddress: "0.0.0.0:0", // TODO
FunctionName: GetenvWithDefault("AWS_LAMBDA_FUNCTION_NAME", "test_function"),
FunctionVersion: functionVersion,

RuntimeInfo: interop.RuntimeInfo{
ImageJSON: "{}",
Arn: "",
Version: ""},
CustomerEnvironmentVariables: additionalFunctionEnvironmentVariables,
SandboxType: interop.SandboxClassic,
Bootstrap: bs,
EnvironmentVariables: env.NewEnvironment(),
}, timeout*1000)
initEnd := time.Now()
return initStart, initEnd
Expand Down
6 changes: 4 additions & 2 deletions cmd/aws-lambda-rie/http.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,18 @@ import (
"net/http"

log "github.com/sirupsen/logrus"
"go.amzn.com/lambda/interop"
"go.amzn.com/lambda/rapidcore"
)

func startHTTPServer(ipport string, sandbox Sandbox) {
func startHTTPServer(ipport string, sandbox *rapidcore.SandboxBuilder, bs interop.Bootstrap) {
srv := &http.Server{
Addr: ipport,
}

// Pass a channel
http.HandleFunc("/2015-03-31/functions/function/invocations", func(w http.ResponseWriter, r *http.Request) {
InvokeHandler(w, r, sandbox)
InvokeHandler(w, r, sandbox.LambdaInvokeAPI(), bs)
})

// go routine (main thread waits)
Expand Down
Loading