AttachVolume.Attach failed for volume "xyz" after node image update

[GeertvanHorrik]

What happened:

Azure AKS applied an auto-update for the node pool. Then the pods are not coming up with this message:

AttachVolume.Attach failed for volume "xyz" : rpc error: code = NotFound desc = failed to get azure instance id for node "aks-default-12345678-vmss000002" (instance not found)

I tried:

1. Restarting the csi-azuredisk-node pod,
2. Deleting PVC and PV, then redeploy the app

PV and PVC are successfully bound after redeploying the app, but the pods cannot attach to the volume.

What you expected to happen:

After an update, the disks should be able to be found and be attachable.

How to reproduce it:

This has happened a few times now. Sometimes we can delete the PVC and PV and redeploy our app, but it's a burden to do this whenever Azure automatically updates the node pool images.

Anything else we need to know?:

Using AKS with managed disks.

Environment:

• CSI Driver version: v1.29.4-1
• Kubernetes version (use kubectl version): 1.28.9

[GeertvanHorrik]

A cluster reboot (stop + start) solves the issue:

Logs:

AttachVolume.Attach failed for volume "xyz" : CSINode aks-default-12345678-vmss000008 does not contain driver disk.csi.azure.com

Then after a while (when the csi pods are up):

AttachVolume.Attach succeeded for volume "xyz"

But restarting an AKS cluster is tricky (takes a long time, no guarantee that there will be resources available to start it again, etc). What would be a better way to solve this without a full cluster restart?

[andyzhangx]

aks-default-12345678-vmss000002 is not up after node pool auto upgrade, that's the reason why disk attach failed in the beginning, does this issue always happen or mitigated automatically after a while? @GeertvanHorrik

[GeertvanHorrik]

aks-default-12345678-vmss000002 is not up after node pool auto upgrade, that's the reason why disk attach failed in the beginning

We waited for more than 8 hours, but our cluster was down all the time (so that's not really an option). We tried scaling with nodes (adding, removing, manually draining, etc). Only when we stopped / started the cluster, it was all good.

[GeertvanHorrik]

It is happening again. 15 minutes ago the agent pool was updated with the latest image. Now it's not finding the disks.

Pods are being scheduled on aks-default-xxxxxxx-vmss000008.

Then getting this error from the driver:

AttachVolume.Attach failed for volume "xxx-pv" : rpc error: code = NotFound desc = failed to get azure instance id for node "aks-default-xxxxxxx-vmss000008" (instance not found)

[GeertvanHorrik]

Cluster reboot immediately fixes the issue (but there is no guarantee we can start the cluster again after stopping it, so it's not an ideal solution).

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[GeertvanHorrik]

This is still an issue.

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[GeertvanHorrik]

It happened again last Sunday.

[GeertvanHorrik]

/remove-lifecycle rotten