Advice: Is there a way of sampling a % of spans

[amccague]

Component(s)

pkg/ottl

Describe the issue you're reporting

I have some noisy spans that I want to filter. But equally I want to know if they're still being produced and from where.

Ideally I'd implement a filter with a percentage, so 1% of the spans still get through. Though I can't see a rand function or something like modulo, though I could possibly craft it using divide. In the absence of rand I could try to use something like the last digit component as my random source.

This doesn't seem like a novel idea though so potentially it's a solved problem and my searching skills aren't up to scatch.

These are leaf spans so it wouldn't impact any children.
I don't think this is worthy use of the tail sampling filter either.

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mkromkamp]

Bot the tailsampling and the probabilistic sampling processor support this scenario I think.

There is a short explanation when to choose which over here. Based on the ottl label I suspect the tail sampler might be the better fit 🙂

Hope that helps.

[amccague]

Hi @mkromkamp - I have reviewed those two processors and to be honest I can't quite determine from the configuration examples in the readme how to achieve this. They appear to sample traces first and foremost.

I can't see the specific configurations to target a span by say attribute value or name. Could you be more specific about the configuration or samplers you had in mind?

Thanks

[mkromkamp]

Hi @mkromkamp - I have reviewed those two processors and to be honest I can't quite determine from the configuration examples in the readme how to achieve this. They appear to sample traces first and foremost.

I can't see the specific configurations to target a span by say attribute value or name. Could you be more specific about the configuration or samplers you had in mind?

Thanks

Yeah sure. I might need a concrete example. But # Rule 2: low sampling for readiness/liveness probes sounds roughly like what you are potentially looking for.

It keeps 10% (0.1) percent of traces for the /live and /ready endpoints for services with the name service-1, service-2, and service-3

Reformatted snippet to avoid having to search

name: team_a-probe
type: and
and:
  and_sub_policy:
    - name: service-name-policy
      type: string_attribute
      string_attribute:
        key: service.name
        values:
          - service-1
          - service-2
          - service-3
    - name: route-live-ready-policy
      type: string_attribute
      string_attribute:
        key: http.route
        values:
          - /live
          - /ready
        enabled_regex_matching: true
    - name: probabilistic-policy
      type: probabilistic
      probabilistic:
        sampling_percentage: 0.1

[amccague]

@mkromkamp

This would as I understand it sample the whole (sub)trace for that route. Per my original request I want to probability sample a particular span, I want to keep the rest of the spans for that given route.