[Test] E2e Tests for Notebook Examples

[Electronic-Waste]

What you would like to be added?

We plan to add e2e tests for notebooks in CI/CD, run with papermill.

REF: https://github.com/nteract/papermill

The notebook examples in need of e2e tests are listed in the following table:

Under examples/v1beta1/kubeflow-pipelines:

• early-stopping.ipynb
• kubeflow-e2e-mnist.ipynb

Under examples/v1beta1/sdk:

• cmaes-and-resume-policies.ipynb
• nas-with-darts.ipynb
• tune-train-from-func.ipynb

And I would strongly recommend that we start with examples under sdk subdirectory since examples under kubeflow-pipelines need full kubeflow components on test env and are more difficult :)

Why is this needed?

This will help us ensure the correctness of our notebook examples for data scientists.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[andreyvelich]

/area testing
/good-first-issue
/remove-label lifecycle/needs-triage

[google-oss-prow]

@andreyvelich:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/area testing
/good-first-issue
/remove-label lifecycle/needs-triage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yehudit1987]

I am interested in solving this
/assign

[gjyotin305]

Hi I am interested in adding this feature, could you tell me which notebooks should I start with ? @Electronic-Waste @andreyvelich

/assign

[Electronic-Waste]

Sorry @gjyotin305, this issue has already been taken by @yehudit1987.

I'm glad to help you if you pick another issue:)

[gjyotin305]

Sorry @gjyotin305, this issue has already been taken by @yehudit1987.

I'm glad to help you if you pick another issue:)

Ok

[Electronic-Waste]

Hi @yehudit1987, are you still working on this?

[andreyvelich]

@Electronic-Waste As you can see @saileshd1402 worked on the E2Es for Notebooks as part of this PR: kubeflow/training-operator#2274.
Can we consolidate effort to build consistent solution for Katib and Training Operator ?
Maybe we can start with Training Operator, and after it setup the same workflow for Katib.

[Electronic-Waste]

@andreyvelich I agree with you. I'll look into the CI for Training Operator first.

And also, this is a kind reminder that PR #2437 is blocked by this issue. Looking forward to your earliest contributions @yehudit1987. I'm happy to help you with any questions:)