Nodes never uncordoned #691

arcdigital · 2024-10-21T22:26:15Z

Image I'm using:
v1.4.0

Issue:
We're seeing our nodes get stuck with the taint node.kubernetes.io/unschedulable:NoSchedule after an update. It doesn't look like this happens to every node when it updates, only some of them. When it does hit this issue - the node successfully performs the update, reboots, but is never uncordoned when it comes back up.

The last thing in the controller logs is the event RebootedIntoUpdate and I see the node reports the new version of bottlerocket

status: Some(BottlerocketShadowStatus { current_version: \"1.24.0\", target_version: \"1.25.0\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None })

Last event I can see in the agent logs is Bottlerocket node is terminated by reboot signal

** Helm Values **

scheduler_cron_expression: "0 0 8 * * Sun"
logging:
  formatter: json
prometheus:
  controller:
    serviceMonitor:
      enabled: true

The text was updated successfully, but these errors were encountered:

cbgbt · 2024-10-30T17:09:05Z

Thanks for raising this issue. The logs you posted are curious:

current_version: \"1.24.0\", target_version: \"1.25.0\", current_state: StagedAndPerformedUpdate

This indicates that the node is currently running 1.24.0 but it has successfully staged the new update, though it has not yet rebooted into it and reported the success back.

The agent logs having their last message be about reboot seems to corroborate this: we triggered the reboot, and Bottlerocket terminated us and started the reboot... but then we never hear from the agent daemon again.

My suspicion is that the brupop agent is never actually restarted on the node after it reboots.

Can you confirm that the node actually rebooted? You could try using kubectl to get the status of the node, or for debugging purposes, use the control or admin containers on bottlerocket and run apiclient get os
Could you query the state of the pods on one of these hosts after the reboot? I would anticipate that the daemonset pods do not have an issue coming back up despite the cordon.

arcdigital · 2024-10-30T17:35:03Z

hey @cbgbt, unfortunately I don't have the logs anymore...but I do remember the node was reporting as running the new version (1.25.0) and I checked the uptime and that seemed to corroborate that too.

cbgbt · 2024-10-30T18:02:44Z

Thank you! I'll attempt to replicate, but if you do happen to encounter this again logs from the agent after the reboot would be very helpful.

arcdigital · 2024-11-14T17:37:31Z

Hey @cbgbt, so the upgrade happened on Nov 3 - The scheduler is set to scheduler_cron_expression: "0 0 8 * * Sun", so ~1am PST.

Current state of the cluster is:
node1 - 1.25.0 - Ready
node2 - 1.25.0 - Ready
node3 - 1.26.1 - node.kubernetes.io/unschedulable:NoSchedule

It looks like the agent pod did not get restarted on the node, but other pods were able to start.

On node3 these pods are scheduled

aws-node (running)
brupop-agent (not running)
ebs-csi-node (not running)
prometheus-node-exporter (ready)
kube-proxy (ready)
our logging agent (ready)
our observability agent (ready)

The agent pod is showing the event "Pod sandbox changed, it will be killed and re-created." every couple of minutes since the node was restarted.

brupop-agent
Created 2024-11-03T01:02:50-07:00
Message Pod sandbox changed, it will be killed and re-created.
Reason SandboxChanged
First seen 2024-11-03T01:02:50-07:00)
Last seen 2024-11-14T09:27:54-08:00)
Count 75246

node3 admin container

[root@admin]# apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "943d9a41",
    "pretty_name": "Bottlerocket OS 1.26.1 (aws-k8s-1.29)",
    "variant_id": "aws-k8s-1.29",
    "version_id": "1.26.1"
  }
}

[root@admin]# uptime
 17:24:09 up 11 days,  9:21,

Most Recent Log for node3 agent

{"timestamp":"2024-11-03T07:57:05.542969Z","level":"INFO","fields":{"message":"Checking for Bottlerocket updates."},"target":"agent::agentclient","span":{"shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","name":"shadow_status_with_refreshed_system_matadata"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(15), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:34:09Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"226005121\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"Idle","name":"update_status_in_shadow"},{"shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","name":"shadow_status_with_refreshed_system_matadata"}]}
{"timestamp":"2024-11-03T07:59:09.239592Z","level":"INFO","fields":{"message":"Checking for Bottlerocket updates."},"target":"agent::agentclient","span":{"shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","name":"shadow_status_with_refreshed_system_matadata"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(15), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:34:09Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"226005121\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"Idle","name":"update_status_in_shadow"},{"shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","name":"shadow_status_with_refreshed_system_matadata"}]}
{"timestamp":"2024-11-03T08:00:04.440951Z","level":"INFO","fields":{"message":"Detected drift between spec state and current state. Requesting node to take action","brs_name":"Some(\"brs-ip-10-50-9-192.ec2.internal\")","action":"StagedAndPerformedUpdate"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227079894\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227079894\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-03T08:00:04.441618Z","level":"INFO","fields":{"message":"Preparing update"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227079894\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227079894\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-03T08:00:24.540863Z","level":"INFO","fields":{"message":"Performing update"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227079894\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227079894\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-03T08:00:55.142790Z","level":"INFO","fields":{"message":"Brs status has been updated.","brs_name":"\"ip-10-50-9-192.ec2.internal\"","brs_status":"BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }"},"target":"agent::agentclient","span":{"name":"update_metadata_shadow"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227079894\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"StagedAndPerformedUpdate","name":"update_status_in_shadow"},{"name":"update_metadata_shadow"}]}
{"timestamp":"2024-11-03T08:01:00.144044Z","level":"INFO","fields":{"message":"Detected drift between spec state and current state. Requesting node to take action","brs_name":"Some(\"brs-ip-10-50-9-192.ec2.internal\")","action":"RebootedIntoUpdate"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-03T08:01:00.144133Z","level":"INFO","fields":{"message":"Rebooting node to complete update"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-03T08:02:21.339337Z","level":"INFO","fields":{"message":"Bottlerocket node is terminated by reboot signal"},"target":"agent::apiclient::api","span":{"args":"[\"raw\", \"-u\", \"/actions/reboot\", \"-m\", \"POST\"]","name":"invoke_apiclient"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},{"name":"reboot"},{"args":"[\"raw\", \"-u\", \"/actions/reboot\", \"-m\", \"POST\"]","name":"invoke_apiclient"}]}

Controller Log

{"timestamp":"2024-10-21T17:40:00.902156Z","level":"INFO","fields":{"message":"starting 4 workers"},"target":"actix_server::builder"}
{"timestamp":"2024-10-21T17:40:00.902270Z","level":"INFO","fields":{"message":"Nothing to do: The bottlerocket-update-operator is not aware of any BottlerocketShadow objects. Is the bottlerocket-shadow CRD installed? Are nodes labelled so that the agent is deployed to them? See the project's README for more information."},"target":"controller::controller"}
{"timestamp":"2024-10-21T17:40:00.902370Z","level":"INFO","fields":{"message":"Actix runtime found; starting in Actix runtime"},"target":"actix_server::server"}
{"timestamp":"2024-10-21T17:42:04.975702Z","level":"INFO","fields":{"message":"BottlerocketShadow has reached desired status. Modifying spec.","desired_spec":"BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-10-21T17:42:04.975697703+00:00\"), version: Some(\"1.25.0\") }"},"target":"controller::controller","span":{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(12), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:39:35Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:04Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187609\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-10-21T17:39:35.985179524+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"},"spans":[{"active_set":"{\"brs-ip-10-50-1-157.ec2.internal\": BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(12), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:39:35Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:04Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187609\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-10-21T17:39:35.985179524+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }}","name":"progress_active_set"},{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(12), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:39:35Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:04Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187609\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-10-21T17:39:35.985179524+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"}]}
{"timestamp":"2024-10-21T17:42:11.008430Z","level":"INFO","fields":{"message":"BottlerocketShadow has reached desired status. Modifying spec.","desired_spec":"BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-10-21T17:42:11.008425484+00:00\"), version: Some(\"1.25.0\") }"},"target":"controller::controller","span":{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(13), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:42:04Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:10Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187714\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-10-21T17:42:04.975697703+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"},"spans":[{"active_set":"{\"brs-ip-10-50-1-157.ec2.internal\": BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(13), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:42:04Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:10Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187714\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-10-21T17:42:04.975697703+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }}","name":"progress_active_set"},{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(13), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:42:04Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:10Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187714\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-10-21T17:42:04.975697703+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"}]}
{"timestamp":"2024-10-21T17:42:25.028342Z","level":"INFO","fields":{"message":"Sleeping until next scheduled time point.","next_schedule_time":"2024-10-27T08:00:00Z","sleep":483454},"target":"controller::scheduler"}
{"timestamp":"2024-10-27T08:00:00.000566Z","level":"INFO","fields":{"message":"Found associated bottlerocketshadow name.","associated_bottlerocketshadow_name":"\"brs-ip-10-50-5-152.ec2.internal\""},"target":"controller::controller","span":{"name":"get_associated_bottlerocketshadow_name"},"spans":[{"name":"find_and_update_ready_brs"},{"name":"get_associated_bottlerocketshadow_name"}]}
{"timestamp":"2024-10-27T08:00:00.000905Z","level":"INFO","fields":{"message":"BottlerocketShadow has reached desired status. Modifying spec.","desired_spec":"BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }"},"target":"controller::controller","span":{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(14), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:42:11Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:23Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187835\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-10-21T17:42:11.008425484+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"},"spans":[{"name":"find_and_update_ready_brs"},{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(14), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"70767658-f5eb-422c-be1c-d2b4f9956239\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:42:11Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-10-21T17:42:23Z)) }]), name: Some(\"brs-ip-10-50-1-157.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-1-157.ec2.internal\", uid: \"70767658-f5eb-422c-be1c-d2b4f9956239\" }]), resource_version: Some(\"220187835\"), self_link: None, uid: Some(\"256422cf-e7fa-400c-945a-27c69d02e7bd\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-10-21T17:42:11.008425484+00:00\"), version: Some(\"1.25.0\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.25.0\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"}]}
{"timestamp":"2024-10-27T08:00:00.018847Z","level":"INFO","fields":{"message":"Began updating new node.","name":"brs-ip-10-50-1-157.ec2.internal"},"target":"controller::controller"}
{"timestamp":"2024-10-27T08:00:02.020658Z","level":"INFO","fields":{"message":"Sleeping until next scheduled time point.","next_schedule_time":"2024-11-03T08:00:00Z","sleep":604797},"target":"controller::scheduler"}
{"timestamp":"2024-11-03T08:00:00.000693Z","level":"INFO","fields":{"message":"Found associated bottlerocketshadow name.","associated_bottlerocketshadow_name":"\"brs-ip-10-50-5-152.ec2.internal\""},"target":"controller::controller","span":{"name":"get_associated_bottlerocketshadow_name"},"spans":[{"name":"find_and_update_ready_brs"},{"name":"get_associated_bottlerocketshadow_name"}]}
{"timestamp":"2024-11-03T08:00:00.000878Z","level":"INFO","fields":{"message":"BottlerocketShadow has reached desired status. Modifying spec.","desired_spec":"BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }"},"target":"controller::controller","span":{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(15), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:34:09Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"226005121\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"},"spans":[{"name":"find_and_update_ready_brs"},{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(15), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-10-21T17:34:09Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-01T08:41:47Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"226005121\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"}]}
{"timestamp":"2024-11-03T08:00:00.019366Z","level":"INFO","fields":{"message":"Began updating new node.","name":"brs-ip-10-50-9-192.ec2.internal"},"target":"controller::controller"}
{"timestamp":"2024-11-03T08:00:56.062865Z","level":"INFO","fields":{"message":"BottlerocketShadow has reached desired status. Modifying spec.","desired_spec":"BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }"},"target":"controller::controller","span":{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080209\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"},"spans":[{"active_set":"{\"brs-ip-10-50-9-192.ec2.internal\": BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080209\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }}","name":"progress_active_set"},{"node":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(16), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:00Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080209\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: StagedAndPerformedUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:00.000875863+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"progress_node"}]}

I tried deleting the agent pod on node3, and after it was recreated it's now saying

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b75dfb54c9cc32d0ac781eeac2be931b11e1af557fe43014e9b5bc00136323e9": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

The aws-node pod seems to be running fine on node3 (we're running vpc-cni v1.18.5-eksbuild.1)

I manually uncordoned the node, and the agent pod successfully started (log below). What's interesting is this does not consistently happen. When I uncordoned the node, the operator started the upgrade on another node which rebooted and was uncordoned successfully. It then did another node, which is stuck cordoned again.

{"timestamp":"2024-11-14T17:44:28.588393Z","level":"INFO","fields":{"message":"Created K8s API Server client using service port","service_port":"443"},"target":"apiserver::client::webclient"}
{"timestamp":"2024-11-14T17:44:29.771842Z","level":"INFO","fields":{"message":"Detected drift between spec state and current state. Requesting node to take action","brs_name":"Some(\"brs-ip-10-50-9-192.ec2.internal\")","action":"RebootedIntoUpdate"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-14T17:44:29.771960Z","level":"INFO","fields":{"message":"Rebooting node to complete update"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-14T17:44:30.482357Z","level":"INFO","fields":{"message":"Checking for Bottlerocket updates."},"target":"agent::agentclient","span":{"shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","name":"shadow_status_with_refreshed_system_matadata"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"RebootedIntoUpdate","name":"update_status_in_shadow"},{"shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","name":"shadow_status_with_refreshed_system_matadata"}]}
{"timestamp":"2024-11-14T17:44:40.881545Z","level":"INFO","fields":{"message":"Brs status has been updated.","brs_name":"\"ip-10-50-9-192.ec2.internal\"","brs_status":"BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }"},"target":"agent::agentclient","span":{"name":"update_metadata_shadow"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"RebootedIntoUpdate","name":"update_status_in_shadow"},{"name":"update_metadata_shadow"}]}
{"timestamp":"2024-11-14T17:44:41.310853Z","level":"INFO","fields":{"message":"Brs status has been updated.","brs_name":"\"ip-10-50-9-192.ec2.internal\"","brs_status":"BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }"},"target":"agent::agentclient","span":{"name":"update_metadata_shadow"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(17), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-03T08:00:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-03T08:00:56Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"227080213\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: RebootedIntoUpdate, state_transition_timestamp: Some(\"2024-11-03T08:00:56.062862950+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.25.0\", target_version: \"1.26.1\", current_state: StagedAndPerformedUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"RebootedIntoUpdate","name":"update_status_in_shadow"},{"name":"update_metadata_shadow"}]}
{"timestamp":"2024-11-14T17:44:46.311806Z","level":"INFO","fields":{"message":"Detected drift between spec state and current state. Requesting node to take action","brs_name":"Some(\"brs-ip-10-50-9-192.ec2.internal\")","action":"MonitoringUpdate"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(18), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:40Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:40Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343489\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-11-14T17:44:40.823163892+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(18), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:40Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:40Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343489\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-11-14T17:44:40.823163892+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-14T17:44:46.311908Z","level":"INFO","fields":{"message":"Ensuring that the node is uncordoned"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(18), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:40Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:40Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343489\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-11-14T17:44:40.823163892+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(18), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:40Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:40Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343489\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-11-14T17:44:40.823163892+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-14T17:44:46.497312Z","level":"INFO","fields":{"message":"Monitoring node's healthy condition"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(18), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:40Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:40Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343489\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-11-14T17:44:40.823163892+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(18), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:40Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:40Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343489\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-11-14T17:44:40.823163892+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-14T17:44:47.003756Z","level":"INFO","fields":{"message":"Brs status has been updated.","brs_name":"\"ip-10-50-9-192.ec2.internal\"","brs_status":"BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }"},"target":"agent::agentclient","span":{"name":"update_metadata_shadow"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(18), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:40Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:40Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343489\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: MonitoringUpdate, state_transition_timestamp: Some(\"2024-11-14T17:44:40.823163892+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: RebootedIntoUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"MonitoringUpdate","name":"update_status_in_shadow"},{"name":"update_metadata_shadow"}]}
{"timestamp":"2024-11-14T17:44:52.005262Z","level":"INFO","fields":{"message":"Detected drift between spec state and current state. Requesting node to take action","brs_name":"Some(\"brs-ip-10-50-9-192.ec2.internal\")","action":"Idle"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(19), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:46Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:48Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343572\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-11-14T17:44:48.845439843+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(19), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:46Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:48Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343572\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-11-14T17:44:48.845439843+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-14T17:44:52.005343Z","level":"INFO","fields":{"message":"Ready to finish monitoring and start update process"},"target":"agent::agentclient","span":{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(19), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:46Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:48Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343572\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-11-14T17:44:48.845439843+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(19), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:46Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:48Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343572\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-11-14T17:44:48.845439843+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","name":"handle_state_transition"}]}
{"timestamp":"2024-11-14T17:44:59.981690Z","level":"INFO","fields":{"message":"Brs status has been updated.","brs_name":"\"ip-10-50-9-192.ec2.internal\"","brs_status":"BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }"},"target":"agent::agentclient","span":{"name":"update_metadata_shadow"},"spans":[{"name":"run"},{"bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2024-09-22T20:20:53Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(19), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:status\": Object {\".\": Object {}, \"f:crash_count\": Object {}, \"f:current_state\": Object {}, \"f:current_version\": Object {}, \"f:target_version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: Some(\"status\"), time: Some(Time(2024-11-14T17:44:46Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object {\"f:metadata\": Object {\"f:ownerReferences\": Object {\".\": Object {}, \"k:{\\\"uid\\\":\\\"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\\\"}\": Object {}}}, \"f:spec\": Object {\".\": Object {}, \"f:state\": Object {}, \"f:state_transition_timestamp\": Object {}, \"f:version\": Object {}}})), manager: Some(\"unknown\"), operation: Some(\"Update\"), subresource: None, time: Some(Time(2024-11-14T17:44:48Z)) }]), name: Some(\"brs-ip-10-50-9-192.ec2.internal\"), namespace: Some(\"addon-bottlerocket-update-operator\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-50-9-192.ec2.internal\", uid: \"c03a8bb8-e7b7-4f9e-ba29-cf040f7bb7b7\" }]), resource_version: Some(\"233343572\"), self_link: None, uid: Some(\"b0bcdddc-dd1c-4119-86b2-abd58344e97f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: Some(\"2024-11-14T17:44:48.845439843+00:00\"), version: Some(\"1.26.1\") }, status: Some(BottlerocketShadowStatus { current_version: \"1.26.1\", target_version: \"1.26.1\", current_state: MonitoringUpdate, crash_count: 0, state_transition_failure_timestamp: None }) }","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","state":"Idle","name":"update_status_in_shadow"},{"name":"update_metadata_shadow"}]}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nodes never uncordoned #691

Nodes never uncordoned #691

arcdigital commented Oct 21, 2024 •

edited

Loading

cbgbt commented Oct 30, 2024 •

edited

Loading

arcdigital commented Oct 30, 2024

cbgbt commented Oct 30, 2024

arcdigital commented Nov 14, 2024 •

edited

Loading

Nodes never uncordoned #691

Nodes never uncordoned #691

Comments

arcdigital commented Oct 21, 2024 • edited Loading

cbgbt commented Oct 30, 2024 • edited Loading

arcdigital commented Oct 30, 2024

cbgbt commented Oct 30, 2024

arcdigital commented Nov 14, 2024 • edited Loading

On node3 these pods are scheduled

The agent pod is showing the event "Pod sandbox changed, it will be killed and re-created." every couple of minutes since the node was restarted.

node3 admin container

Most Recent Log for node3 agent

Controller Log

arcdigital commented Oct 21, 2024 •

edited

Loading

cbgbt commented Oct 30, 2024 •

edited

Loading

arcdigital commented Nov 14, 2024 •

edited

Loading