🐛clustercatalog_controller: hacky, but more correct status updating in reconciler #424

joelanford · 2024-10-03T20:20:00Z

For awhile in catalogd, we've had a very loose "needsUnpack" behavior where we would make the assumption that if we didn't need to unpack, then we also didn't need to update the status of the object.

This means that some objects may never have their status updated after an upgrade. That's problematic because upgrades may add new helpful fields or fix bugs in the way that status is reported.

This PR is a somewhat hacky fix for the situation. It solves the problem by keeping an in-memory map of the observedGeneration and resolved source, and then using that as the source of truth to and then use that expected status as part of the determination of whether an unpack is required.

A new criteria of "needsUnpack" is now: "does the expected status differ from the status that is on the object. If so, unpack again.

Fixes #422
Fixes #420

codecov · 2024-10-03T20:22:53Z

Codecov Report

Attention: Patch coverage is 85.10638% with 14 lines in your changes missing coverage. Please review.

Project coverage is 38.55%. Comparing base (8137da0) to head (63b0914).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rnal/controllers/core/clustercatalog_controller.go	87.20%	11 Missing ⚠️
cmd/manager/main.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #424      +/-   ##
==========================================
+ Coverage   35.28%   38.55%   +3.26%     
==========================================
  Files          14       14              
  Lines         802      843      +41     
==========================================
+ Hits          283      325      +42     
- Misses        472      474       +2     
+ Partials       47       44       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

joelanford · 2024-10-03T20:27:26Z

internal/controllers/core/clustercatalog_controller.go

+	Unpacker source.Unpacker
+	Storage  storage.Instance
+
+	finalizers crfinalizer.Finalizers


I needed to refactor finalizers so that it had access to the reconciler struct. That way it would be possible to delete(r.storedCatalogs, catalog.Name) in the finalizer logic.

joelanford · 2024-10-03T20:28:24Z

internal/controllers/core/clustercatalog_controller.go

+	case r.needsPoll(storedCatalog.unpackResult.ResolvedSource.Image.LastSuccessfulPollAttempt.Time, catalog):
+		l.Info("unpack required: poll duration has elapsed")
+		needsUnpack = true
+	}


I intentionally added lots of logging here to help us deduce why we might be unpacking more frequently than expected.

joelanford · 2024-10-03T20:30:15Z

internal/controllers/core/clustercatalog_controller.go

+	clearUnknownConditions(expectedStatus)
+	if hasStoredCatalog && r.Storage.ContentExists(catalog.Name) {
+		updateStatusServing(expectedStatus, storedCatalog.unpackResult, r.Storage.ContentURL(catalog.Name), storedCatalog.observedGeneration)
+		updateStatusProgressing(expectedStatus, storedCatalog.observedGeneration, nil)


I needed to change the updateStatusProgressing signature so that I could call it with a deep-copied status, and a different observed generation.

bentito · 2024-10-03T23:25:59Z

internal/source/containers_image_test.go

@@ -396,7 +396,7 @@ func TestImageRegistry(t *testing.T) {
 				// If the digest should already exist check that we actually hit it
 				if tt.digestAlreadyExists {
 					assert.Contains(t, buf.String(), "image already unpacked")
-					assert.Equal(t, rs.UnpackTime, unpackDirStat.ModTime())
+					assert.Equal(t, rs.UnpackTime, unpackDirStat.ModTime().Truncate(time.Second))


I think even asserting on the second could be brittle if we happen to run on the second to next second break. How about just asserting the duration is in a reasonable range?

Suggested change

assert.Equal(t, rs.UnpackTime, unpackDirStat.ModTime().Truncate(time.Second))

assert.WithinDuration(t, rs.UnpackTime, unpackDirStat.ModTime(), time.Second)

This one should be exact. rs.UnpackTime is always truncated to the second in the same way. I'll add a comment in the code about why we need that though.

bentito · 2024-10-03T23:54:45Z

@joelanford when I point current operator-controller main at this with a replace and run make test-upgrade-e2e I am still seeing:

# github.com/operator-framework/operator-controller/internal/catalogmetadata/cache
internal/catalogmetadata/cache/cache.go:86:42: catalog.Status.ResolvedSource.Image.ResolvedRef undefined (type *"github.com/operator-framework/catalogd/api/core/v1alpha1".ResolvedImageSource has no field or method ResolvedRef)
internal/catalogmetadata/cache/cache.go:122:62: catalog.Status.ResolvedSource.Image.ResolvedRef undefined (type *"github.com/operator-framework/catalogd/api/core/v1alpha1".ResolvedImageSource has no field or method ResolvedRef)
internal/catalogmetadata/cache/cache.go:164:52: catalog.Status.ResolvedSource.Image.ResolvedRef undefined (type *"github.com/operator-framework/catalogd/api/core/v1alpha1".ResolvedImageSource has no field or method ResolvedRef)
# github.com/operator-framework/operator-controller/internal/controllers
internal/controllers/clusterextension_controller.go:419:53: oldObject.Status.ResolvedSource.Image.ResolvedRef undefined (type *"github.com/operator-framework/catalogd/api/core/v1alpha1".ResolvedImageSource has no field or method ResolvedRef)
internal/controllers/clusterextension_controller.go:419:106: newObject.Status.ResolvedSource.Image.ResolvedRef undefined (type *"github.com/operator-framework/catalogd/api/core/v1alpha1".ResolvedImageSource has no field or method ResolvedRef)
# github.com/operator-framework/operator-controller/internal/controllers [github.com/operator-framework/operator-controller/internal/controllers.test]
internal/controllers/clusterextension_controller.go:419:53: oldObject.Status.ResolvedSource.Image.ResolvedRef undefined (type *"github.com/operator-framework/catalogd/api/core/v1alpha1".ResolvedImageSource has no field or method ResolvedRef)
internal/controllers/clusterextension_controller.go:419:106: newObject.Status.ResolvedSource.Image.ResolvedRef undefined (type *"github.com/operator-framework/catalogd/api/core/v1alpha1".ResolvedImageSource has no field or method ResolvedRef)
make: *** [vet] Error 1

joelanford · 2024-10-04T00:26:52Z

@bentito, that's expected. See the accompanying changes in operator-framework/operator-controller#1329.

No matter how you slice it, there is a breaking change from 0.29.0 to 0.30.0 (and from 0.29.0 to this PR):

We removed some status fields that operator-controller depends on, and
We changed the spec.source.type enum from image to Image, with no backcompat support. So the upgrade tests are bound to fail because the new catalogd no longer understands old data (i.e. spec.source.type: "image")

bentito · 2024-10-04T11:57:12Z

@bentito, that's expected. See the accompanying changes in operator-framework/operator-controller#1329.

No matter how you slice it, there is a breaking change from 0.29.0 to 0.30.0 (and from 0.29.0 to this PR):
1. We removed some status fields that operator-controller depends on, and

2. We changed the `spec.source.type` enum from `image` to `Image`, with no backcompat support. So the upgrade tests are bound to fail because the new catalogd no longer understands old data (i.e. `spec.source.type: "image"`)

ah, okay, understanding that now, was just worried it was supposed to be fixed from catalogd side alone

ankitathomas

Looks good! Left a small comment about the status comparison

ankitathomas · 2024-10-04T13:48:51Z

internal/controllers/core/clustercatalog_controller.go

+	clearUnknownConditions(expectedStatus)
+	if hasStoredCatalog && r.Storage.ContentExists(catalog.Name) {
+		updateStatusServing(expectedStatus, storedCatalog.unpackResult, r.Storage.ContentURL(catalog.Name), storedCatalog.observedGeneration)
+		updateStatusProgressing(expectedStatus, storedCatalog.observedGeneration, nil)


Is this to avoid waiting till the next poll before reconciling in case of unpack errors? Looks like the equivalence check for the Progressing condition will always fail if we had an error on the last unpack; do we backoff between reconciling errored unpacks?

Oh this is a good question. Here's a scenario:

We are successfully serving content for generation 1

We reconcile generation 2, which has a bad image ref, let's say the tag doesn't exist. So we:

Do not update storedCatalog map

DO update the status with Progressing=True, Retrying

On the next reconcile (that is requeued basically immediately), we build an "expectedStatus" that says Installed=True and Progressing=False based on this code. We later compare this to the real status from the object and we are told that there are differences, which forces an unpack.

Step 3 is a little odd though. Maybe if we return an error from Reconcile we should delete(r.storedCatalogs, catalog.Name) as a better signal for "we need to attempt to unpack again"?

ankitathomas · 2024-10-04T14:09:40Z

internal/controllers/core/clustercatalog_controller.go

+			updateStatusProgressing(&catalog.Status, catalog.GetGeneration(), storageErr)
 			return ctrl.Result{}, storageErr


considering this isn't a terminal error, should we requeue here?

By returning the storageErr, we get requeue based on exponential backoff. We only want the requeue based on poll interval if there are no errors during reconcile.

… reconciler Signed-off-by: Joe Lanford <[email protected]>

joelanford requested a review from a team as a code owner October 3, 2024 20:20

joelanford commented Oct 3, 2024

View reviewed changes

bentito reviewed Oct 3, 2024

View reviewed changes

joelanford force-pushed the fix-needs-unpack branch from 797ff05 to 2a05670 Compare October 4, 2024 00:22

ankitathomas previously approved these changes Oct 4, 2024

View reviewed changes

ankitathomas reviewed Oct 4, 2024

View reviewed changes

LalatenduMohanty previously approved these changes Oct 4, 2024

View reviewed changes

everettraven previously approved these changes Oct 4, 2024

View reviewed changes

clustercatalog_controller: hacky, but more correct status updating in…

63b0914

… reconciler Signed-off-by: Joe Lanford <[email protected]>

joelanford dismissed stale reviews from everettraven, LalatenduMohanty, and ankitathomas via 63b0914 October 4, 2024 20:34

joelanford force-pushed the fix-needs-unpack branch from 2a05670 to 63b0914 Compare October 4, 2024 20:34

everettraven approved these changes Oct 4, 2024

View reviewed changes

joelanford enabled auto-merge October 4, 2024 20:40

joelanford added this pull request to the merge queue Oct 4, 2024

Merged via the queue into operator-framework:main with commit dee3337 Oct 4, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛clustercatalog_controller: hacky, but more correct status updating in reconciler #424

🐛clustercatalog_controller: hacky, but more correct status updating in reconciler #424

joelanford commented Oct 3, 2024 •

edited

Loading

codecov bot commented Oct 3, 2024 •

edited

Loading

joelanford Oct 3, 2024 •

edited

Loading

joelanford Oct 3, 2024

joelanford Oct 3, 2024

bentito Oct 3, 2024

joelanford Oct 3, 2024

bentito commented Oct 3, 2024

joelanford commented Oct 4, 2024

bentito commented Oct 4, 2024

ankitathomas left a comment

ankitathomas Oct 4, 2024 •

edited

Loading

joelanford Oct 4, 2024 •

edited

Loading

ankitathomas Oct 4, 2024 •

edited

Loading

joelanford Oct 4, 2024

	assert.Equal(t, rs.UnpackTime, unpackDirStat.ModTime().Truncate(time.Second))
	assert.WithinDuration(t, rs.UnpackTime, unpackDirStat.ModTime(), time.Second)

		updateStatusProgressing(&catalog.Status, catalog.GetGeneration(), storageErr)
		return ctrl.Result{}, storageErr

🐛clustercatalog_controller: hacky, but more correct status updating in reconciler #424

🐛clustercatalog_controller: hacky, but more correct status updating in reconciler #424

Conversation

joelanford commented Oct 3, 2024 • edited Loading

codecov bot commented Oct 3, 2024 • edited Loading

Codecov Report

joelanford Oct 3, 2024 • edited Loading

Choose a reason for hiding this comment

joelanford Oct 3, 2024

Choose a reason for hiding this comment

joelanford Oct 3, 2024

Choose a reason for hiding this comment

bentito Oct 3, 2024

Choose a reason for hiding this comment

joelanford Oct 3, 2024

Choose a reason for hiding this comment

bentito commented Oct 3, 2024

joelanford commented Oct 4, 2024

bentito commented Oct 4, 2024

ankitathomas left a comment

Choose a reason for hiding this comment

ankitathomas Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

joelanford Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

ankitathomas Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

joelanford Oct 4, 2024

Choose a reason for hiding this comment

joelanford commented Oct 3, 2024 •

edited

Loading

codecov bot commented Oct 3, 2024 •

edited

Loading

joelanford Oct 3, 2024 •

edited

Loading

ankitathomas Oct 4, 2024 •

edited

Loading

joelanford Oct 4, 2024 •

edited

Loading

ankitathomas Oct 4, 2024 •

edited

Loading