Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support running e2e tests on AWS #45

Draft
wants to merge 43 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
61ef090
support running e2e tests on aws
eaudetcobello Aug 19, 2024
24bbde4
add ami id
eaudetcobello Aug 19, 2024
9c3c084
hacks to get the cluster deployed
eaudetcobello Aug 21, 2024
a4f07d2
newest template updates
eaudetcobello Aug 21, 2024
4d85a76
add cluster-templates
bschimke95 Jul 8, 2024
d82fca1
Add clusterctl config
bschimke95 Jul 8, 2024
448d4e6
docs
bschimke95 Jul 10, 2024
57aa7bc
move clusterctl and update template-variables
bschimke95 Jul 11, 2024
53b39bc
delete obsolete docs
bschimke95 Jul 12, 2024
4c6e40f
overwrite install scriptes
bschimke95 Jul 16, 2024
8d7e872
update template configuration
eaudetcobello Aug 15, 2024
e19d3f5
configurable ccm image
eaudetcobello Aug 15, 2024
45ab4de
remove clusterctl
eaudetcobello Aug 21, 2024
7d52c3a
review comments
eaudetcobello Aug 21, 2024
b303aaf
reformat
eaudetcobello Aug 21, 2024
3022a2b
remove microk8s references
eaudetcobello Aug 21, 2024
aaac27d
add ami id to .rc file
eaudetcobello Aug 21, 2024
53468c7
use nodename field, ccm label
eaudetcobello Aug 21, 2024
1c3677b
configurable root volume size. fix var. name
eaudetcobello Aug 21, 2024
9c61f28
remove hardcoded namespace
eaudetcobello Aug 21, 2024
23f5a94
proper e2e config
eaudetcobello Aug 24, 2024
923b3cf
revert add aws pkg
eaudetcobello Aug 24, 2024
ea1fc5f
newline at eof
eaudetcobello Aug 24, 2024
a8f55f6
add info to readme
eaudetcobello Aug 28, 2024
1fbcf1f
add region env. var
eaudetcobello Aug 28, 2024
d86460d
explain how to run the tests with tilt
eaudetcobello Aug 28, 2024
7fb852b
add the ability to run tests on aws
eaudetcobello Aug 29, 2024
054f0f0
Merge branch 'main' into KU-1266/e2e-test-aws
eaudetcobello Aug 29, 2024
9ee8c47
Add running cluster in CI
eaudetcobello Sep 3, 2024
9a99562
reset e2e.yaml
eaudetcobello Sep 3, 2024
fcdff45
Remove new file docs/overview.md
eaudetcobello Sep 3, 2024
45b5e97
Make install script more robust
eaudetcobello Sep 4, 2024
c2cefe5
Add Github action
eaudetcobello Sep 4, 2024
cdee3ec
can run tests locally in lxd container
eaudetcobello Sep 5, 2024
0f2f9c5
only pass env. vars in script step
eaudetcobello Sep 5, 2024
3320f66
add debug action
eaudetcobello Sep 9, 2024
8019c0a
delete file
eaudetcobello Sep 12, 2024
e889508
update comments
eaudetcobello Sep 12, 2024
c95520b
test
eaudetcobello Sep 25, 2024
c6ea395
Merge branch 'main' into KU-1266/e2e-test-aws
eaudetcobello Sep 25, 2024
c4846b9
cleanup ck8s-aws.yaml
eaudetcobello Sep 26, 2024
012f061
don't use custom script
eaudetcobello Sep 26, 2024
8aeee09
launch test on aws
eaudetcobello Sep 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions .github/workflows/e2e-deleteme.yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused about the name of this file, I guess the contents of this file was supposed to be carried over to the e2e.yaml file or something

Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: E2E Tests

on:
pull_request:

permissions:
contents: read
id-token: write

jobs:
run-e2e-tests:
name: Run E2E Tests
runs-on: ubuntu-latest
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Setup Go
uses: actions/setup-go@v2
with:
go-version: ~1.22.0

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
audience: sts.amazonaws.com
aws-region: us-east-2
role-to-assume: arn:aws:iam::018302341396:role/GithubOIDC
role-duration-seconds: 3600

- name: Check out repo
uses: actions/checkout@v4

- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
with:
detached: true

- name: Install requirements
run: |
sudo apt install make wget
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.24.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

- name: Increase inotify watches
run: |
# Prevents https://cluster-api.sigs.k8s.io/user/troubleshooting#cluster-api-with-docker----too-many-open-files
sudo sysctl fs.inotify.max_user_watches=1048576
sudo sysctl fs.inotify.max_user_instances=8192

- name: Build images
run: |
cd hack
./build-e2e-images.sh
cd ..
make docker-build-e2e

- name: Run e2e tests
run: |
E2E_INFRA=aws GINKGO_FOCUS="Workload cluster creation" make test-e2e
env:
GOPROXY: direct
2 changes: 2 additions & 0 deletions .github/workflows/e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ name: E2E Tests

on:
pull_request:
branches:
- does-not-exist
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and I guess after moving the content of deleteme file to this file, we should remove this line


permissions:
contents: read
Expand Down
9 changes: 7 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -92,14 +92,15 @@ KUSTOMIZE := $(TOOLS_BIN_DIR)/$(KUSTOMIZE_BIN)-$(KUSTOMIZE_VER)
# Ginkgo
TEST_DIR := $(shell pwd)/test
ARTIFACTS ?= $(shell pwd)/_artifacts
GINKGO_FOCUS ?=
GINKGO_FOCUS ?= Workload cluster creation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is a left over from testing?

Copy link
Contributor Author

@eaudetcobello eaudetcobello Sep 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is temporary. I am leaving it like this for now because this test passes on all infras and it's easier to iterate with one test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by temporary do we mean that it's gonna get removed before this PR is merged? or are we planning to change this in the up coming PRs? if it's the latter, maybe we can add a note or todo.
also I think this will change the scope for both the docker and aws tests, right? I think we only need this for the aws test.

GINKGO_SKIP ?=
GINKGO_NODES ?= 1 # GINKGO_NODES is the number of parallel nodes to run
# when running the e2e tests, 1 means no parallelism
GINKGO_TIMEOUT ?= 2h
GINKGO_POLL_PROGRESS_AFTER ?= 60m
GINKGO_POLL_PROGRESS_INTERVAL ?= 5m
E2E_CONF_FILE ?= $(TEST_DIR)/e2e/config/ck8s-docker.yaml
E2E_INFRA ?= aws
E2E_CONF_FILE ?= $(TEST_DIR)/e2e/config/ck8s-$(E2E_INFRA).yaml
SKIP_RESOURCE_CLEANUP ?= false
USE_EXISTING_CLUSTER ?= false
GINKGO_NOCOLOR ?= false
Expand Down Expand Up @@ -407,3 +408,7 @@ $(CONTROLLER_GEN): ## Build controller-gen from tools folder.

$(CONVERSION_GEN): ## Build conversion-gen from tools folder.
GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) k8s.io/code-generator/cmd/conversion-gen $(CONVERSION_GEN_BIN) $(CONVERSION_GEN_VER)

.PHONY: nuke
nuke:
aws-nuke -c ~/.config/aws-nuke/nuke-config.yaml --no-dry-run
96 changes: 96 additions & 0 deletions hack/aws-nuke-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
regions:
- us-east-2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can extract this from this file and also the workflow file. I think this might cause subtle problems if we duplicate it.


blocklist:
- "999999999999" # A blocklist is required, but this is a placeholder

accounts:
018302341396: # AWS Account IDs are not considered sensitive information
filters:
EC2Instance:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2SecurityGroup:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2Volume:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2InternetGateway:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2NATGateway:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2RouteTable:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2Subnet:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2VPC:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2VPCEndpoint:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2VPCEndpointServiceConfiguration:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2ElasticIP:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
EC2NetworkInterface:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
ELBv2:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
ELBv2TargetGroup:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
S3Bucket:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
AutoScalingGroup:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
AutoScalingLaunchConfiguration:
- property: "tag:sigs.k8s.io/cluster-api-provider-aws/cluster/capick8s-*"
type: "glob"
invert: true
resource-types:
includes:
- EC2Instance
- EC2SecurityGroup
- EC2Volume
- EC2InternetGateway
- EC2NATGateway
- EC2RouteTable
- EC2Subnet
- EC2VPC
- EC2VPCEndpoint
- EC2VPCEndpointServiceConfiguration
- EC2ElasticIP
- EC2NetworkInterface
- ELBv2
- ELBv2TargetGroup
- S3Bucket
- AutoScalingGroup
- AutoScalingLaunchConfiguration
Loading
Loading