Feature Request: Bundle testscases and resources by feature

[viccuad]

Is your feature request related to a problem?

Right now, the secure supply chain tests have:

• The testcases at https://github.com/kubewarden/kubewarden-end-to-end-tests/blob/main/tests/secure-supply-chain-tests.bats
• The needed resources at:
  • https://github.com/kubewarden/kubewarden-end-to-end-tests/blob/main/resources/restricted-secure-supply-chain-verification-config.yaml
  • https://github.com/kubewarden/kubewarden-end-to-end-tests/blob/main/resources/secure-supply-chain-verification-config.yaml

Solution you'd like

I would like to have a folder similar to:

tests
└── supply-chain
    ├── e2e.bats
    └── resources
        ├── config1.yml
        ├── config2.yml
        └── policy.yml

So:

• it's easier to ascertain what relates to what. It also helps for finding examples of policies and configs when one is debugging or implementing new features.
• Changing a resource for a test doesn't break other unrelated tests.
• Cleanup of unneeded resources is easier.

This would mean that instead of hard-coding which tests are run here, the runner/makefile should just recursively check the tests folder for .bats files and execute them.

Bonus points if the little recursive script skips folders whose name end in .skip (which allows to disable portions of the test as needed when developing), supports several levels of directories (so one can have tests under supply-chain/policies-install, supply-chain/policy, supply-chain/policy/sbom, supply-chain/policy/images) and works fine no matter where in the tree it is called.

If the entry points for tests is not an executable ending in .bats but just an executable, one could use different testing tools in different folders as needed (maybe bats is nice to test one feature, but venom for another).

Alternatives you've considered

No response

Anything else?

No response