clean up all resources for job when job is suspended

[KunWuLuan]

Controller will clean up all resources for job when job is suspended

[cheyang]

Why not using isSuspend to avoid creating resources like pod? If it's for ECI scenario, will it create pod and delete it after a while?

[KunWuLuan]

Thanks for your time and assistance. @cheyang

We will check the job status here:

tf-operator/pkg/controller.v1/tensorflow/controller.go

Line 431 in b81dfd7

if isSucceeded(tfjob.Status) || isFailed(tfjob.Status) || tfJobExceedsLimit {

and we will start to reconcile pods only when the checking failed:

tf-operator/pkg/controller.v1/tensorflow/controller.go

Line 496 in b81dfd7

err = tc.reconcilePods(tfjob, pods, rtype, spec, replicasStatus)

So we don't need another pre checking out of reconcileTFJobs.