[Microsoft.Extensions.Http.Resilience] Circuit-breaker: shared policy, CustomCircuitBreakerStateProvider per client?

[natwallbank]

Hey all,

I'm hoping this might be the correct place to post my query, as I note a number of other questions around the new resilience library. I'm just getting started with it, and what I was hoping to achieve was the following:

• Define a set of named policies (via AddResiliencePipeline)
• Be able to reference these named policies for created HttpClients via AddResilienceHandler (it's likely that all of our HttpClients will use the same policy in the first instance, but it gives us the flexibility to introduce others as we develop our service)

Those two steps seem straightforward, but where I'm a little confused is around the use of CircuitBreakerStateProvider and CircuitBreakerManualControl. I'd like to be able to use these on a per-client basis so that I can view the state of the circuit for each dependency, and control it manually for testing purposes.

The library doesn't seem to be set up for this though, unless I'm missing something obvious, as you have to supply instances of those two types when configuring your CircuitBreakerStrategyOptions as part of your pipeline specification.

Am I missing something obvious? Thanks in advance!

(I guess the obvious solution is to define a policy per-client and share the strategies, but it intuitively feels like it goes against the point of defining policies separately - but maybe that's my misunderstanding)

[iliar-turdushev]

Hello @natwallbank,

If understood you correctly then you were trying to create a single shared resilience pipeline but to have its CircuitBreakerStrategy being distinct per each named HttpClient, right? If so then such a scenario is not supported out of the box, and I would say is not standard. I would personally prefer the approach you suggested define a policy per-client and share the strategies:

// Clients "default" and "custom" share resilience strategies, but each has its own instance of the strategies.
// In particular, each client has its own CircuitBreaker, StateProvider, and ManualControl.
services.AddHttpClient("default").AddResilienceHandler("default-client", DefaultPipeline);
services.AddHttpClient("custom").AddResilienceHandler("custom-client", DefaultPipeline);

public static void DefaultPipeline(ResiliencePipelineBuilder<HttpResponseMessage> builder)
{
    builder.AddRetry(...).AddCircuitBreaker(...)...
}

You can also achieve the same with the approach you described shared policy, CircuitBreaker per client but it would be too verbose. You can use something like below:

// We'll use complex key PipelineKey build pipelines:
// - for each PipelineName we'll have one pipeline builder;
// - for each combination PipelineName+ClientName we'll have a single pipeline.
// For more info see https://www.pollydocs.org/pipelines/resilience-pipeline-registry.html.
services.Configure<ResiliencePipelineRegistryOptions<PipelineKey>>(options =>
{
    options.BuilderNameFormatter = key => key.PipelineName;
    options.InstanceNameFormatter = key => key.ClientName;
    options.BuilderComparer = PipelineKey.BuilderComparer;
});

// Here we register a pipeline. `ClientName` doesn't matter here, you can put here any value.
// Only `PipelineName` matters, since it defines the name of the pipeline.
services.AddResiliencePipeline<PipelineKey>(new("pipeline", "default"), (builder, context) =>
{
    builder.AddRetry(...).AddCircuitBreaker(...)...
});

services.AddHttpClient("default").AddResilienceHandler("default-client", (builder, context) =>
{
    var provider = context.ServiceProvider.GetRequiredService<ResiliencePipelineProvider<PipelineKey>>();
    // Here we create a pipeline "pipeline" for a client "default".
    builder.AddPipeline(provider.GetPipeline(new PipelineKey("pipeline", "default")));
});

services.AddHttpClient("custom").AddResilienceHandler("custom-client", (builder, context) =>
{
    var provider = context.ServiceProvider.GetRequiredService<ResiliencePipelineProvider<PipelineKey>>();
    // Here we create a pipeline "pipeline" for a client "custom". This pipeline is different from the client "default".
    // Both pipelines "default" and "custom" have their own CircuitBreaker.
    builder.AddPipeline(provider.GetPipeline(new PipelineKey("pipeline", "custom")));
});

----------

public readonly record struct PipelineKey(string PipelineName, string ClientName)
{
    public static readonly IEqualityComparer<PipelineKey> BuilderComparer = new BuilderEqualityComparer();

    private sealed class BuilderEqualityComparer : IEqualityComparer<PipelineKey>
    {
        public bool Equals(PipelineKey x, PipelineKey y) => StringComparer.Ordinal.Equals(x.PipelineName, y.PipelineName);

        public int GetHashCode(PipelineKey obj) => StringComparer.Ordinal.GetHashCode(obj.PipelineName);
    }
}

Does it answer your questions?

[natwallbank]

It does, thanks. I was just seeking clarification that I wasn't missing something obvious in the API design.

I've already gone ahead and implemented a pipeline per client to sort manual circuit-breaking, so quite relieved I won't need any rework 🙂.

Thanks again!