Internal feature: Stateless scheduler

[Omrigan]

Problem description / Motivation

Currently, the scheduler acts as the source of truth for the cluster state. We'd like to change that because of the following reasons:

1. Makes scheduler replication hard, as it is unclear how to reliably replicate that state.
2. Every scheduler restart results in a state reset, thus a period of warm-up is needed to repopulate the state by the autoscaler-agent. This mechanism is responsible for additional complication of the components logic and protocol, namely:
   a. The existence of the Buffer values.
   b. The notion of lastPermit.
3. Potentially, the upscaling communication diagram can be simplified to bypass the autoscaler-agent after the upscaling was approved by scheduler.

Feature idea(s) / DoD

1. A scheduler can be replicated.
2. A scheduler restart doesn't cause the sub-optimal scheduling decisions.
3. We no longer have the concepts of Buffered resources and lastPermit.

Implementation ideas

The scheduler's state and autoscaler-agent ↔ scheduler protocol can be moved entirely to annotations on the associated VirtualMachine object.

Related work

The implementation of neondatabase/neon#8111 might allow us to significantly simplify autoscaler-agent (or even merge it's remaining functionality into other components).

Further information

There are more details in the internal RFC.

[sharnoff]

Status update: I've written a draft implementation, doesn't yet pass tests.

This week, I'll finish up that initial implementation, and write up an RFC/design doc.