XFS Ephemeral Storage fails to mount with: Superblock has unknown incompatible features (0x20) enabled.

[snowzach]

Image I'm using: bottlerocket-aws-k8s-1.25-x86_64-v1.26.1-943d9a41
This is on a i3en.2xlarge node with local NVMe flash I'm hoping to use as Ephemeral storage.
If I set the type to ext4 it works.

What I expected to happen: The local flash storage would be formatted and mounted as XFS storage

What actually happened: The storage isn't mounted.

I'm guessing because the xfs tools are newer than the kernel?

Error in dmesg:

[  302.626669] XFS (md127): Superblock has unknown incompatible features (0x20) enabled.
[  302.626673] XFS (md127): Filesystem cannot be safely mounted by this kernel.
[  302.626675] XFS (md127): SB validate failed with error -22.

How to reproduce the problem:

Put this in the config:

[settings.bootstrap-commands.k8s-ephemeral-storage]
    commands = [
      ["apiclient", "ephemeral-storage", "init", "-t", "xfs"],
      ["apiclient", "ephemeral-storage" ,"bind", "--dirs", "/var/lib/kubelet"]
    ] 
    essential = true
    mode = "always"

[yeazelm]

Hello @snowzach, thanks for cutting this issue!

I believe this is due to the level of XFS support in the 5.15 kernel used in the k8s 1.25 variant. We validated XFS with the 6.1 kernel (all k8s variants 1.28+ and ECS 2 variants) for ephemeral storage, but not on the older variants. We use XFS as the default data partition on these same variants so we know it works well with the 6.1 kernel. We'll need to dig into what the issue might be with the 5.15 kernel and see if we can fix it.

As you found, for ephemeral storage commands, ext4 should work fine on k8s 1.25 and XFS should work well on k8s 1.28+, are you able to switch to a newer k8s variant for XFS support while we sort out what is going on with XFS on k8s 1.25?

[yeazelm]

I was able to get a mountable xfs volume for the 5.15 kernel by using this command:

mkfs.xfs -f -m bigtime=1,inobtcount=1 -i nrext64=0 /dev/md127

The only thing strictly required here is the -i nrext64=0. We would need to add logic to the format command to properly format the filesystem when on 5.15 kernels to make this work. This would enable XFS usage on 5.15 kernels with ephemeral storage.

[snowzach]

Thanks for looking into it! Unfortunately, I'm not going to be able to upgrade K8S for a bit.

[cbgbt]

No worries. I'm working on applying changes to the module @yeazelm mentioned which should do the right thing on older kernels by avoiding nrext64, which is a new XFS feature added in 6.1.