Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend liveness probe timing #156

Conversation

bshephar
Copy link
Collaborator

@bshephar bshephar commented Jul 3, 2023

The Horizon pod is fairly frequently killed due to the current liveness probe timing. This seems to happen more frequently during the initial deployment while the system is under load. This change extends the liveness probe to provide some extra time for kubelet to validate the functionality of the Horizon pod before terminating it.

@openshift-ci openshift-ci bot requested review from abays and viroel July 3, 2023 01:51
@openshift-ci openshift-ci bot added the approved label Jul 3, 2023
@bshephar bshephar force-pushed the extend-liveness-check branch 2 times, most recently from be40c02 to 74d33e0 Compare July 3, 2023 02:15
@abays
Copy link
Contributor

abays commented Jul 3, 2023

The Horizon pod is fairly frequently killed due to the current liveness probe timing. This seems to happen more frequently during the initial deployment while the system is under load. This change extends the liveness probe to provide some extra time for kubelet to validate the functionality of the Horizon pod before terminating it.

If this failure happens during the initial deployment, I wonder if it would be prudent to consider adding a startup probe [1]?

[1] https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes

The Horizon pod is fairly frequently killed due to the current liveness
probe timing. This seems to happen more frequently during the initial
deployment while the system is under load. This change adds the
startup probe to provide some extra time for kubelet to validate the
functionality of the Horizon pod before terminating it.

Signed-off-by: Brendan Shephard <[email protected]>
@bshephar
Copy link
Collaborator Author

bshephar commented Jul 3, 2023

The Horizon pod is fairly frequently killed due to the current liveness probe timing. This seems to happen more frequently during the initial deployment while the system is under load. This change extends the liveness probe to provide some extra time for kubelet to validate the functionality of the Horizon pod before terminating it.

If this failure happens during the initial deployment, I wonder if it would be prudent to consider adding a startup probe [1]?

[1] https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes

Sounds reasonable. Let's give it a run and see how it goes. I changed my commit to reflect this.

Copy link
Contributor

@abays abays left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Jul 3, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 3, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, bshephar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit fb6d1d7 into openstack-k8s-operators:main Jul 3, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants