Trigger ETL Lambda

A lambda function that takes the data extracted from the getJIRAData project, transforms it using AWS Glue and writes the output to an S3 bucket to be picked up by a subsequent job to ingest into AWS QuickSight for visualisation.

This is part 2 of an overall solution to provide DevOps Key Performance Indicators on a AWS QuickSight dashboard.

1. Retrieve the data
2. Transfor the data and perform the KPI calculation (This repository))
3. Ingest the transformed data to QuickSight - Coming shortly

Build and deploy

1. Clone the repository

2. Install the following dependencies

   1. jq
   2. AWS CLI (v2.0.62)

3. ./sh package.sh

   (Creates the triggerETL.zip file in the build folder)

4. ./sh upload.sh

   (Copies the zip file to the specified S3 bucket to store the lambda function)

5. ./sh create-stack.sh

   (Creates the CloudFormation stack for all of the required resources)

6. ./sh upload-etl.sh

   (Copies the ETL script used by the Glue job to the S3 etl-scripts bucket)

7. ./sh create-s3-notification.sh

   (Adds the PutObject notification event to the S3 input bucket that is created in part 1 of the solution to trigger this lambda)

8. ./sh start-crawler.sh

   (Runs the Glue crawler to create the table schema from the input data. The input job will need to be run once to provide the input data.)

The following AWS resources are created

1. Lambda function
2. Glue
   1. Crawler
   2. Classifier
   3. Database
   4. Job
3. Role - the execution role associated with the Lambda function to provide access to XRay, SSM, S3 and CF
4. S3 buckets
   1. source for the retrieved JIRA issues.
   2. output for transformed data
   3. ETL script

Architecture Diagram