feat: add nvidia MIG

[piyush-jena]

Issue number:

Related:

• feat: add nvidia MIG Settings bottlerocket-settings-sdk#63
• Support NVIDIA MIG for more secure multi-tenant GPU sharing / allocation bottlerocket#4252

Description of changes:
Adding nvidia-migmanager service and binary that configures the instance with nvidia mig.

Testing done:

1. Instance joined the cluster

NAME                                           STATUS   ROLES    AGE   VERSION
ip-XXXX.us-west-2.compute.internal   Ready    <none>   15h   v1.29.5-eks-1109419

2. Model Default:

bash-5.1# apiclient get settings.kubelet-device-plugin
{
  "settings": {
    "kubelet-device-plugins": {
      "nvidia": {
        "device-id-strategy": "index",
        "device-list-strategy": "volume-mounts",
        "device-partitioning-strategy": "none",
        "device-sharing-strategy": "none",
        "pass-device-specs": true
      }
    }
  }
}

3. Model Updates:

bash-5.1#: apiclient set settings.kubelet-device-plugins.nvidia.device-partitioning-strategy="mig"
bash-5.1#: apiclient set settings.kubelet-device-plugins.nvidia.mig.profile."a100-40gb"="1g.5gb"
bash-5.1# apiclient get settings.kubelet-device-plugin
{
  "settings": {
    "kubelet-device-plugins": {
      "nvidia": {
        "device-id-strategy": "index",
        "device-list-strategy": "volume-mounts",
        "device-partitioning-strategy": "mig",
        "device-sharing-strategy": "none",
        "mig": {
          "profile": {
            "a100-40gb": "1g.5gb"
          }
        },
        "pass-device-specs": true
      }
    }
  }
}

kubectl describe node shows 56 gpus post instance reboot.

4. Bounded check:

bash-5.1# apiclient set settings.kubelet-device-plugins.nvidia.mig.profile."a100-40gb"="1g.7gb"
Failed to change settings: Failed PATCH request to '/settings/keypair?tx=apiclient-set-oAusAKGk9ClUpd44': Status 400 when PATCHing /settings/keypair?tx=apiclient-set-oAusAKGk9ClUpd44: Unable to match your input to the data model. We may not have enough type information. Please try the —json input form. Cause: Error during deserialization: Unable to deserialize into MIGProfile: Invalid MIG Profile value '1g.7gb' at line 1 column 76
bash-5.1#

5. Files generated:

bash-5.1# cat /etc/nvidia-k8s-device-plugin/settings.yaml
version: v1
flags:
  migStrategy: "single"
  failOnInitError: true
  plugin:
    passDeviceSpecs: true
    deviceListStrategy: volume-mounts
    deviceIDStrategy: index

bash-5.1# cat etc/nvidia-migmanager/nvidia-migmanager.toml
device-partitioning-strategy = "mig"
profile = { a100-40gb = "1g.5gb" }

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.