Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backend] Incorrect run status reporting for Runs using ExitHandler #11405

Open
DharmitD opened this issue Nov 25, 2024 · 2 comments
Open

[backend] Incorrect run status reporting for Runs using ExitHandler #11405

DharmitD opened this issue Nov 25, 2024 · 2 comments

Comments

@DharmitD
Copy link
Contributor

Description

Pipeline overall run status inaccurately reports Succeeded when using an ExitHandler that includes a failing task, but succeeding exit_task

Suggested solution and more context can be found here. In short, we should try to first attempt to use Argo lifecycle hooks when creating the dag task for the exit_task.

Steps to reproduce

To reproduce this, use the following attached pipeline.
exithandler-v2-yaml.txt

Expected result

pipeline run should report accurate status if a task in an exit handler task group fails.

Materials and Reference


Impacted by this bug? Give it a 👍.

@DharmitD
Copy link
Contributor Author

/assign @DharmitD

@mprahl
Copy link
Contributor

mprahl commented Dec 17, 2024

/assign @mprahl

mprahl added a commit to mprahl/pipelines that referenced this issue Dec 17, 2024
As described in kubeflow#10917, exit handlers were implemented as dependent
tasks that always ran within an Argo Workflow. The issue is that this
caused the pipeline to have a succeeded status regardless of if the
tasks within the exit handlers all succeeded.

This commit changes exit handlers to be exit lifecycle hooks on an
Argo Workflow so that the overall pipeline status is not impacted.

Resolves:
kubeflow#11405

Signed-off-by: mprahl <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants