Support running e2e tests on AWS #45

eaudetcobello · 2024-08-19T19:07:14Z

ci-e2e-tests.sh → my goal is to make this script a one-step solution to run the e2e tests, it launches a lxc container on the local machine and all the necessary tools are installed into it, and then tests are run within. It requires AWS credential configuration variables to be set on the local machine.
e2e-deleteme.yaml → temporary file I created to not interefere with the existing workflows that are defined in e2e.yaml. End goal is that whatever is in e2e-deleteme.yaml should be merged with e2e.yaml and a test matrix should be used that includes each architecture.
aws-nuke-config.yaml → file is still a WIP. It’s the config. to use with aws-nuke. The goal is for this config to cleanup all resources that have a certain tag. I’m still evaluating if this is the best way to go about things. Because of the way aws-nuke is built, we need to pass an exact tag, which means we can’t use a glob to include all capi resources (like tag:sigs.k8s.io/cluster-api/cluster/*). We need to say specifically say tag:sigs.k8s.io/cluster-api/cluster/capick8s-test-2djn3. One way to circumvent this would be to grep the cluster name from the output of the e2e test logs, then use sed to replace the tag in the aws-nuke configuration file.
install-aws-nuke.sh, install-clusterawsadm.sh, install-clusterctl.sh → install these tools into the lxc container's /usr/local/bin
write-provider-config.sh → this writes the CAPI provider configuration to the clusterctl.yaml config. file. We need this because ck8s is not a part of the default providers, so we need to tell clusterctl where to find our provider.
cluster_upgrade_test.go → Here we replace the hardcoded reference to docker infrastructure and instead tell the test suite to resolve `. This is then read from the config. file of the test (I think), which lives at test/config/ck8s-{docker,aws}.yaml
create_test.go → same as above.
test/e2e/data/infrastructure-aws folder → this folder contains the templates that will be applied to the cluster during the test. cluster-template.yaml is the one I have tested “Workload cluster creation” test with. The remediation.yaml templates are pretty much the same as the cluster-template.yaml except that they add a MachineHealthCheck and labels under MachineDeployment.
test/e2e/shared/v1beta1 → this defines the major and minor contract version of the core cluster api components that we use for the tests. (See ck8s-aws.yaml line 25)
test/e2e/shared/v1beta1_aws → this defines the major and minor contract version of the CAPA resources we deploy. (See ck8s-aws.yaml Line 38)
kcp_remediation_test.go, md_remediation_test.go, node_scale_test.go → only changed the hardcoded docker provider to clusterctl.DefaultInfrastructureProvider

bschimke95

great work @eaudetcobello, added couple of comments
we should probably get #22 in before this.

bschimke95 · 2024-08-20T07:53:06Z

Makefile

@@ -92,14 +92,14 @@ KUSTOMIZE := $(TOOLS_BIN_DIR)/$(KUSTOMIZE_BIN)-$(KUSTOMIZE_VER)
 # Ginkgo
 TEST_DIR := $(shell pwd)/test
 ARTIFACTS ?= $(shell pwd)/_artifacts
-GINKGO_FOCUS ?=
+GINKGO_FOCUS ?= Workload cluster creation


I guess this is a left over from testing?

Yes, this is temporary. I am leaving it like this for now because this test passes on all infras and it's easier to iterate with one test.

by temporary do we mean that it's gonna get removed before this PR is merged? or are we planning to change this in the up coming PRs? if it's the latter, maybe we can add a note or todo.
also I think this will change the scope for both the docker and aws tests, right? I think we only need this for the aws test.

bschimke95 · 2024-08-20T07:54:50Z

test/e2e/config/ck8s-aws.yaml

+      # when bootstrapping with tilt, it will use
+      # the defaultProviderVersion in https://github.com/kubernetes-sigs/cluster-api/blob/main/hack/tools/internal/tilt-prepare/main.go as
+      # default version for docker infrastructure provider
+      # name here should match defaultProviderVersion


this looks like it does not belong here.

It belongs, but I will clean up the message.

test/e2e/config/ck8s-aws.yaml

test/e2e/create_test.go

Makefile

test/e2e/e2e_suite_test.go

test/e2e/data/infrastructure-aws/cluster-template.yaml

- Change service CIDR block to match our standard range - Create a bastion by default - Copy cluster bootstrap/join overrides to control-plane - Remove unneeded prerunCommands - Remove tolerations that don't apply to our setup (kubernetes.io/master) - Add args from default provider template to aws-cloud-controller-manager args - Restore some RBAC rules from default provider template

no hardcoded defaults, empty variables by default, use new nodeName field to specify node name

support tilt with tilt-provider.json

HomayoonAlimohammadi

Really nice work! Thanks a lot @eaudetcobello! did an initial pass and left some comments.

HomayoonAlimohammadi · 2024-09-19T09:59:39Z

.github/workflows/e2e-deleteme.yaml

+    strategy:
+      matrix:
+        ginkgo_focus:
+          #- "KCP remediation"


do we plan to uncomment these? or are these commented by mistake? maybe we can add a note or todo if this is intentional

HomayoonAlimohammadi · 2024-09-19T10:02:31Z

.github/workflows/e2e-deleteme.yaml

+      - name: Install requirements
+        run: |
+          sudo apt install make
+          sudo apt install wget


maybe we can install everything in a single line

HomayoonAlimohammadi · 2024-09-19T10:12:52Z

.github/workflows/e2e-deleteme.yaml

I'm a bit confused about the name of this file, I guess the contents of this file was supposed to be carried over to the e2e.yaml file or something

HomayoonAlimohammadi · 2024-09-19T10:13:16Z

.github/workflows/e2e.yaml

@@ -2,6 +2,8 @@ name: E2E Tests

 on:
  pull_request:
+    branches:
+      - does-not-exist


and I guess after moving the content of deleteme file to this file, we should remove this line

HomayoonAlimohammadi · 2024-09-19T10:17:19Z

Makefile

@@ -92,14 +92,14 @@ KUSTOMIZE := $(TOOLS_BIN_DIR)/$(KUSTOMIZE_BIN)-$(KUSTOMIZE_VER)
 # Ginkgo
 TEST_DIR := $(shell pwd)/test
 ARTIFACTS ?= $(shell pwd)/_artifacts
-GINKGO_FOCUS ?=
+GINKGO_FOCUS ?= Workload cluster creation


by temporary do we mean that it's gonna get removed before this PR is merged? or are we planning to change this in the up coming PRs? if it's the latter, maybe we can add a note or todo.
also I think this will change the scope for both the docker and aws tests, right? I think we only need this for the aws test.

HomayoonAlimohammadi · 2024-09-19T10:19:23Z

hack/aws-nuke-config.yaml

@@ -0,0 +1,96 @@
+regions:
+  - us-east-2


maybe we can extract this from this file and also the workflow file. I think this might cause subtle problems if we duplicate it.

HomayoonAlimohammadi · 2024-09-19T10:24:33Z

.github/workflows/e2e-deleteme.yaml

+          sudo sysctl fs.inotify.max_user_instances=8192
+      - name: Run e2e tests
+        run: |
+          sudo -E ./hack/ci-e2e-tests.sh true aws v0.1.2


I can't understand this true here, would elaborate a bit? I see ${0}, ${2} and ${3} in the script, this should be ${1}, right?

HomayoonAlimohammadi · 2024-09-19T10:27:53Z

hack/ci-e2e-tests.sh

really nice work putting up this script together. it should have been really challenging. but I strongly feel like we need to change this to a (maybe) python script cause it's too complicated for a bash script. not for this task but in the near future.

eaudetcobello requested a review from a team as a code owner August 19, 2024 19:07

eaudetcobello changed the title ~~Support running e2e tests on aws~~ Support running e2e tests on AWS Aug 19, 2024

bschimke95 reviewed Aug 20, 2024

View reviewed changes

eaudetcobello marked this pull request as draft August 20, 2024 12:51

neoaggelos reviewed Aug 20, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

test/e2e/e2e_suite_test.go Outdated Show resolved Hide resolved

test/e2e/data/infrastructure-aws/cluster-template.yaml Outdated Show resolved Hide resolved

eaudetcobello added 3 commits August 21, 2024 10:47

support running e2e tests on aws

61ef090

add ami id

24bbde4

hacks to get the cluster deployed

9c3c084

eaudetcobello force-pushed the KU-1266/e2e-test-aws branch from c0609c4 to 9c3c084 Compare August 21, 2024 14:47

newest template updates

a4f07d2

eaudetcobello force-pushed the KU-1266/e2e-test-aws branch from b4b0cb7 to a4f07d2 Compare August 21, 2024 18:15

bschimke95 and others added 17 commits August 23, 2024 21:04

add cluster-templates

4d85a76

Add clusterctl config

d82fca1

docs

448d4e6

move clusterctl and update template-variables

57aa7bc

delete obsolete docs

53b39bc

overwrite install scriptes

4c6e40f

configurable ccm image

e19d3f5

remove clusterctl

45ab4de

review comments

7d52c3a

no hardcoded defaults, empty variables by default, use new nodeName field to specify node name

reformat

b303aaf

remove microk8s references

3022a2b

add ami id to .rc file

aaac27d

use nodename field, ccm label

53468c7

configurable root volume size. fix var. name

1c3677b

remove hardcoded namespace

9c61f28

proper e2e config

23f5a94

support tilt with tilt-provider.json

eaudetcobello force-pushed the KU-1266/e2e-test-aws branch from ad3a016 to 23f5a94 Compare August 24, 2024 01:14

revert add aws pkg

923b3cf

eaudetcobello force-pushed the KU-1266/e2e-test-aws branch 19 times, most recently from 190c2a5 to 54dddf0 Compare September 12, 2024 00:09

add debug action

3320f66

eaudetcobello force-pushed the KU-1266/e2e-test-aws branch from 54dddf0 to 3320f66 Compare September 12, 2024 00:10

eaudetcobello added 2 commits September 11, 2024 22:42

delete file

8019c0a

update comments

e889508

HomayoonAlimohammadi reviewed Sep 19, 2024

View reviewed changes

eaudetcobello added 5 commits September 25, 2024 16:13

test

c95520b

Merge branch 'main' into KU-1266/e2e-test-aws

c6ea395

cleanup ck8s-aws.yaml

c4846b9

don't use custom script

012f061

launch test on aws

8aeee09

eaudetcobello force-pushed the KU-1266/e2e-test-aws branch from b9a596d to 8aeee09 Compare September 26, 2024 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support running e2e tests on AWS #45

Support running e2e tests on AWS #45

eaudetcobello commented Aug 19, 2024 •

edited

Loading

bschimke95 left a comment

bschimke95 Aug 20, 2024

eaudetcobello Sep 12, 2024 •

edited

Loading

HomayoonAlimohammadi Sep 19, 2024

bschimke95 Aug 20, 2024

eaudetcobello Sep 12, 2024

HomayoonAlimohammadi left a comment

HomayoonAlimohammadi Sep 19, 2024

HomayoonAlimohammadi Sep 19, 2024

HomayoonAlimohammadi Sep 19, 2024

HomayoonAlimohammadi Sep 19, 2024

HomayoonAlimohammadi Sep 19, 2024

HomayoonAlimohammadi Sep 19, 2024

HomayoonAlimohammadi Sep 19, 2024

HomayoonAlimohammadi Sep 19, 2024

Support running e2e tests on AWS #45

Are you sure you want to change the base?

Support running e2e tests on AWS #45

Conversation

eaudetcobello commented Aug 19, 2024 • edited Loading

bschimke95 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eaudetcobello Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HomayoonAlimohammadi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eaudetcobello commented Aug 19, 2024 •

edited

Loading

eaudetcobello Sep 12, 2024 •

edited

Loading