branchprotector continuously failing

[BenTheElder]

It's timing out after 5h20m: https://prow.k8s.io/?job=ci-test-infra-branchprotector

This is causing us to not sync branch protection rules, e.g. related to #33857 / #33880

We have a big problem with this tooling not scaling, it uses a ton of API quota and needs to run continuously and scales with the number of repos/branches which is only growing.

/area prow
/sig testing
/priority important-soon

[BenTheElder]

filed #33901 as a stopgap. it's not great.

[BenTheElder]

Aborted the current run and started a new re-run with the latest config and the 12h timeout, but we know it will be at least 5+ hours before we can tell if that worked as a stopgap ... https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-test-infra-branchprotector/1864084620887199744

[BenTheElder]

That run failed at 11h35m57s.

The logs are enourmous and full of 404 errors:

protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.2 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.4 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.5 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-7.0 from protected=false: get current branch protection: getting branch protection 404: Not Found, update jobselector from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.3 from protected=true: get current branch protection: getting branch protection 404: Not Found], update kubectl-validate: [update alexzielenski-patch-1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update alexzielenski-patch-2 from protected=true: get current branch protection: getting branch protection 404: Not Found, update apelisse-exit-code-suggestion from protected=true: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update kube-scheduler-wasm-extension: [update golang-sdk from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=true: get current branch protection: getting branch protection 404: Not Found], update sigs-github-actions: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update knftables: update master from protected=true: get current branch protection: getting branch protection 404: Not Found, update noderesourcetopology-api: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update cloud-pv-admission-labeler: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update cluster-inventory-api: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update obscli: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update node-ipam-controller: [update readme from protected=true: get current branch protection: getting branch protection 404: Not Found, update update-codegen/fix from protected=true: get current branch protection: getting branch protection 404: Not Found, update init from protected=false: get current branch protection: getting branch protection 404: Not Found, update e2e_aojea from protected=true: get current branch protection: getting branch protection 404: Not Found, update integration-tests from protected=true: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update lws: [update release-0.2.0 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.4.0 from protected=false: get current branch protection: getting branch protection 404: Not Found, update release-0.4.1 from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.4.2 from protected=false: get current branch protection: getting branch protection 404: Not Found, update release-0.3.0 from protected=true: get current branch protection: getting branch protection 404: Not Found, update rupliu/migrate from protected=true: get current branch protection: getting branch protection 404: Not Found, update rupliu/refactor from protected=true: get current branch protection: getting branch protection 404: Not Found], update testgrid: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update referencegrant-poc: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update karpenter-provider-cluster-api: [update multiplex-client from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update kube-network-policies: [update aojea-patch-3 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-4 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-6 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-7 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-2 from protected=true: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-5 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-8 from protected=false: get current branch protection: getting branch protection 404: Not Found], update secrets-store-sync-controller: [update gh-pages from protected=false: get policy: kubernetes-sigs/secrets-store-sync-controller=gh-pages defines a policy, which requires protect: true, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update wg-device-management: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update etcd-manager: [update fix-release-process-docs from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update cve-feed-osv: [update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2020-10749.json from protected=true: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2024-5321.json from protected=true: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2020-8553.json from protected=false: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2024-7646.json from protected=false: get current branch protection: getting branch protection 404: Not Found], update wg-serving: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update multi-network: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update multi-network-api: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update llm-instance-gateway: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update gwctl: [update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=false: get current branch protection: getting branch protection 404: Not Found], update cni-dra-driver: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update container-object-storage-interface: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update ingate: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update kjob: update main from protected=false: get current branch protection: getting branch protection 404: Not

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-test-infra-branchprotector/1864084620887199744

[BenTheElder]

I think it at least barely completed running without being killed by the timeout, but the error output is massive and it feels like it's not running correctly.

NOTE: this has actually been ongoing for months AFAICT, there have been scattered threads in slack. It's just gotten enough attention now to make sure we have a tracking issue >.<

[BenTheElder]

some discussion but nothing conclusive in https://kubernetes.slack.com/archives/CDECRSC5U/p1733337720772809

[pohly]

This causes the "Waiting for status to be reported" for the removed pull-kubernetes-verify-lint because it is the job of the branch protector to tell GitHub which statuses it should wait for - or in this case, not wait for anymore.

The removal of that job has not been mirrored to GitHub yet, despite that completed run.

[pacoxu]

A quick proposal can be dividing this CI to several CIs group by sig/project/org. Is this doable?

[BenTheElder]

A quick proposal can be dividing this CI to several CIs group by sig/project/org. Is this doable?

That was discussed in the slack thread above. But currently the job is completing, so quota/throughput, while bad, is not the problem.

It's clearly also bugged. Someone will have to spend more time investigating this.

An immediate mitigation is reaching out to github management to manually update github settings, but these problems will continue to impact the organization if the tooling isn't fixed.

[BenTheElder]

Raised in #github-management: https://kubernetes.slack.com/archives/C01672LSZL0/p1733438889444729

[BenTheElder]

@MadhavJivrajani manually removed pull-kubernetes-verify-lint from kubernetes/kubernetes branch protection rules as a stop-gap mitigation. But this system is still fundamentally not working well.

[BenTheElder]

We're still seeing issues with blocking merge requirements not being removed from repos: https://kubernetes.slack.com/archives/C01672LSZL0/p1734188195640379

/help

[k8s-ci-robot]

@BenTheElder:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

We're still seeing issues with blocking merge requirements not being removed from repos: https://kubernetes.slack.com/archives/C01672LSZL0/p1734188195640379

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BenTheElder]

filed kubernetes-sigs/prow#345