option to not auto-add tolerations

[tooptoop4]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

looking at a pod (kubectl get pod -o yaml) for a step in a workflow i see it has these tolerations added:

  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300

this can cause issues (unschedulable) where a step/pod is being assigned to a new node that is not ready ie aws/amazon-vpc-cni-k8s#2808 where it flicks between ready/notready and back to ready

Version

3.5.6

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

n/a

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

n/a

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

n/a

[agilgur5]

I don't think Argo ever sets its own tolerations, only the ones that you specify. There's only one place in the code that they're added too.

Why do you think Argo is setting these?

[tooptoop4]

turns out to be a k8s default https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions

Note:
Kubernetes automatically adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300, unless you, or a controller, set those tolerations explicitly.

These automatically-added tolerations mean that Pods remain bound to Nodes for 5 minutes after one of these problems is detected.

perhaps docs should mention overwriting this

[agilgur5]

perhaps docs should mention overwriting this

Tolerations are only mentioned in the Fields Reference, where they directly inherit from k8s, so have no Argo specific behavior nor Argo specific docs.