The nvme disk mount failure with azureDisk - findDiskByLun(0) failed with error(failed to find disk by lun 0)

[duanwei33]

What happened:
Azure disk mount failed on some instance types with error: azureDisk - findDiskByLun(0) failed with error(failed to find disk by lun 0) in Openshift, currently found the following:
Standard_M16bds_v3
Standard_L8s_v4
Standard_M16bs_v3

$ oc describe pod mypod-test-1
  Warning  FailedMount             <invalid> (x3 over <invalid>)  kubelet                  MountVolume.MountDevice failed for volume "pvc-2bad271a-abce-4ab1-975b-385bd08e4261" : rpc error: code = Internal desc = failed to find disk on lun 0. azureDisk - findDiskByLun(0) failed with error(failed to find disk by lun 0)

When checking the node, seems there is no lun info (not sure if it is the reason):

 lsblk -o NAME,KNAME,MAJ:MIN,FSTYPE,SIZE,TYPE,MOUNTPOINT,HCTL
NAME        KNAME     MAJ:MIN FSTYPE   SIZE TYPE MOUNTPOINT HCTL
nvme0n1     nvme0n1   259:0            128G disk
|-nvme0n1p1 nvme0n1p1 259:1              1M part
|-nvme0n1p2 nvme0n1p2 259:2   vfat     127M part
|-nvme0n1p3 nvme0n1p3 259:3   ext4     384M part /boot
`-nvme0n1p4 nvme0n1p4 259:4   xfs    127.5G part /sysroot
nvme0n2     nvme0n2   259:5             10G disk.                         <------ this is the attached volume.

What you expected to happen:
The disk should be mounted successfully

How to reproduce it:
Always

Anything else we need to know?:

Environment:

• CSI Driver version: 1.31.1
• Kubernetes version (use kubectl version): v1.31.3
• OS (e.g. from /etc/os-release): Red Hat Enterprise Linux CoreOS release 4.18
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[andyzhangx]

it's missing udev rules for nvme controller disks on the node, could you run following command, and then try again?

kubectl apply -f https://raw.githubusercontent.com/andyzhangx/demo/refs/heads/master/aks/download-v6-disk-rules.yaml

[duanwei33]

it's missing udev rules for nvme controller disks on the node, could you run following command, and then try again?

kubectl apply -f https://raw.githubusercontent.com/andyzhangx/demo/refs/heads/master/aks/download-v6-disk-rules.yaml

Yes, it works when I added the 80-azure-disk.rules!
Thank you so much!

[duanwei33]

One more question, should CSI Driver cover this or does it need to be handled by node os part?

[andyzhangx]

we have set up this new udev rules in the new aks vhh image, it would rollout in aks rp 2025.2 release.

[andyzhangx]

o, you are using OpenShift, so I think you should include it in your image by yourself. here is an example PR: Azure/AgentBaker#5444

[duanwei33]

Yeah acked, thanks for the confirmation.
And Happy New Year!