Skip to content
This repository has been archived by the owner on Dec 13, 2023. It is now read-only.

Permissive task capability #3866

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

ivakoleva
Copy link

@ivakoleva ivakoleva commented Nov 27, 2023

Pull Request type

  • Bugfix
  • Feature
  • Refactoring (no functional changes, no api changes)
  • Build related changes (Please run ./gradlew generateLock saveLock to refresh dependencies)
  • WHOSUSING.md
  • Other (please describe):

NOTE: Please remember to run ./gradlew spotlessApply to fix any format violations. (Check.)

Changes in this PR

Why:
We need idempotent forked tasks, meaning all tasks get executed, but any failures are still detected upon join.
Issue #3861

What:
Introduced the concept of Permissive tasks.
A Permissive task is similar to a Simple task. The difference is, it permits the other tasks to continue - in case a Permissive task failed.
Result is:

  1. Forked Permissive tasks will let each other be evaluated, until all the forked tasks had terminated. Only then, the join task should fail. In case of Permissive optional tasks, the join will not fail.
  2. Permissive sequential tasks will let subsequent tasks continue. While at the end, the workflow will fail in case a permissive task had failed. The workflow would not fail in case of Permissive optional task failure.

Testing done:
PermissiveTaskMapperTest added,
TestDeciderOutcomes.testPermissive() added,
WorkflowAndTaskConfigurationSpec "Test simple workflow which has a permissive task" and "Test simple workflow which has a permissive optional task added" that cover retry,
ForkJoinSpec "Test a simple workflow with fork join permissive failure flow" and "Test retrying a failed permissive fork join workflow" added.

In addition, performed e2e tests locally running a Conductor instance. Did build a docker image with the code changes made, started it locally, and started a SampleWorker to poll 3 tasks in parallel. Verified e2e scenarios of task_def_permissive, task_def_permissive_optional, task_def_simple.json, task_def_simple_optional.json, each joining on 6 forked tasks, then running simple task 7 after join.

Alternatives considered

ivakoleva and others added 7 commits November 27, 2023 15:11
Why:
We need idempotent forked tasks, meaning all tasks get
executed, but any failures are still detected upon join.
Feature request Netflix#3861

What:
Introduced the concept of Permissive tasks.
A Permissive task is similar to a Simple task. The
difference is, it permits the other tasks to continue -
in case a Permissive task failed.
Result is:
1. Forked Permissive tasks will let each other
be evaluated, until all the forked tasks had terminated.
Only then, the join task should fail. In case of Permissive
optional tasks, the join will not fail.
2. Permissive sequential tasks will let subsequent tasks
continue. While at the end, the workflow will fail in case
a permissive task had failed. The workflow would not fail
in case of Permissive optional task failure.

Testing done: PermissiveTaskMapperTest added,
TestDeciderOutcomes.testPermissive() added,
WorkflowAndTaskConfigurationSpec
"Test simple workflow which has a permissive task" and
"Test simple workflow which has a permissive optional task added"
that cover retry, ForkJoinSpec
"Test a simple workflow with fork join permissive failure flow"
added.
In addition, performed e2e tests locally running a Conductor instance.
Did build a docker image with the code changes made, started it locally,
and started a SampleWorker to poll 3 tasks in parallel.
Verified e2e scenarios of task_def_permissive, task_def_permissive_optional,
task_def_simple.json, task_def_simple_optional.json, each joining on
6 forked tasks, then running simple task 7 after join.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant