pageserver: assert that uploads don't modify indexed layers

[erikgrinaker]

Problem

It's not legal to modify layers that are referenced by the current layer index. Assert this in the upload queue, as preparation for upload queue reordering.

Touches #10096.

Summary of changes

Add a debug assertion that the upload queue does not modify layers referenced by the current index.

I could be convinced that this should be a plain assertion, but will be conservative for now.

[github-actions]

6450 tests run: 6167 passed, 2 failed, 281 skipped (full report)

---

Failures on Postgres 17

• test_sharding_split_compaction[compact-shard-ancestors-persistent]: debug-x86-64
• test_sharding_split_compaction[None]: debug-x86-64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_sharding_split_compaction[debug-pg17-compact-shard-ancestors-persistent] or test_sharding_split_compaction[debug-pg17-None]"

Flaky tests (2)

Postgres 17

• test_timeline_archival_chaos: release-arm64

Postgres 14

• test_lr_with_slow_safekeeper: release-arm64

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
4399771 at 2024-12-23T13:02:13.058Z :recycle:}

[erikgrinaker]

Looks like we're hitting this in test_sharding_split_compaction:

ERROR remote_upload{tenant_id=d70f77446951ab06ea4cc25667e14468 shard_id=0002 timeline_id=70f0688bdda68b926b34cc72b86f376a upload_task_id=6}:panic{thread=background op worker location=pageserver/src/tenant/remote_timeline_client.rs:1960:25}: layer 000000067F0000000500000B16000000003F-000000067F0000000500000B180000000003__00000000014E8F20 modified while referenced by index

Probably something related to shard splits that isn't taken into account here.