Restrict raidz faulted vdev count

[don-brady]

Motivation and Context

In a ZFS pool, leaf vdevs can be faulted after tripping a diagnostic failure (ZED). However ZFS prevents faulting a drive if it would cause data loss. For example, in a raidz2, faulting more than 2 disks would cause read failures.

In determining if a vdev can be faulted, the check in vdev_dtl_required() doesn't consider the case where a marginal drive is being resilvered. If the marginal disk is considered valid during the checks, then zfs can start encountering read errors when the faulted vdev count matches the parity count.

Description

When a drive replacement is active, the replacing vdev should not be considered when checking if a peer can be faulted. I.e., treat it as if it is offline and cannot contribute. Specifically, a child in a replacing vdev won't count when assessing the dtl during a vdev_fault().

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.

How Has This Been Tested?

1. Added a new test: functional/fault/fault_limits
2. Manual testing

Example result from fault_limits showing that the third request to fault a leaf vdev in a raidz3 group was denied and turned instead to a degrade due to presence of a replacing vdev.

  pool: fault-test-pool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: resilvered 654K in 00:00:02 with 0 errors on Wed Sep 25 17:27:33 2024
config:

        NAME                  STATE     READ WRITE CKSUM
        fault-test-pool       DEGRADED     0     0     0
          raidz3-0            DEGRADED     0     0     0
            /var/tmp/dev-0    FAULTED      0     0     0  external device fault
            /var/tmp/dev-1    FAULTED      0     0     0  external device fault
            /var/tmp/dev-2    DEGRADED     0     0     0  external device fault
            /var/tmp/dev-3    ONLINE       0     0     0
            /var/tmp/dev-4    ONLINE       0     0     0
            /var/tmp/dev-5    ONLINE       0     0     0
            /var/tmp/dev-6    ONLINE       0     0     0
            /var/tmp/dev-7    ONLINE       0     0     0
            replacing-8       DEGRADED     0     0     0
              /var/tmp/dev-8  ONLINE       0     0     0
              /var/tmp/dev-9  OFFLINE      0     0     0

errors: No known data errors

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

@don-brady can you rebase this on the latest master code. We cut over to using GitHub Actions fully for the CI and this change needs to include those commits so it can run the tests.

[don-brady]

Rebased to latest master branch

[mcmilk]

It looked right before the ZTS system was running, but afterwards - sth. does not work as intended I think :/

[don-brady]

Fixed the failing test cases. Turns out that the original fix was breaking vdev replace.

[allanjude]

Fixed the failing test cases. Turns out that the original fix was breaking vdev replace.

Thanks Don