Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datadog Receiver spans don't follow OTel specs #36924

Open
cyrille-leclerc opened this issue Dec 23, 2024 · 1 comment
Open

Datadog Receiver spans don't follow OTel specs #36924

cyrille-leclerc opened this issue Dec 23, 2024 · 1 comment
Labels
bug Something isn't working needs triage New item requiring triage receiver/datadog

Comments

@cyrille-leclerc
Copy link
Member

cyrille-leclerc commented Dec 23, 2024 •
edited
Loading

Component(s)

receiver/datadog

What happened?

Description

Spans emitted by Datadog APM agents and converted by the OTel Collector Datadog Receiver don't comply with OTel Semantic conventions, creating challenges to visualize them.

Misalignment applying to all span types

  • A resource attribute deployment.resource.name should be defined instead of the span attribute deployment.environment: Str(production)
  • A resource attribute process.pid should be defined instead of the attribute process.id

Misalignment applying to HTTP server span

Specs: https://opentelemetry.io/docs/specs/semconv/http/http-spans/

Example Datadog HTTP Server span produced by a Spring Boot app (code) and converted by the OTel Col Datadog receiver

2024-12-23T11:28:04.619+0100	info	ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> service.version: Str(1.1)
     -> telemetry.sdk.version: Str(Datadog-1.44.1~13a9a2d011)
     -> telemetry.sdk.name: Str(Datadog)
     -> telemetry.sdk.language: Str(java)
     -> process.runtime.version: Str(17.0.13)
     -> service.name: Str(shipping)
     -> host.name: Str(localhost)
     -> os.type: Str(darwin)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope Datadog 1.44.1~13a9a2d011
Span #0
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 288018d171bc9c67
    ID             : 3e65de74bd03a50a
    Name           : servlet.request
    Kind           : Server
    Start time     : 2024-12-23 10:28:03.717953709 +0000 UTC
    End time       : 2024-12-23 10:28:03.778773334 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(POST /shipOrder)
     -> sampling.priority: Str(1.000000)
     -> datadog.span.id: Str(4496244395968013578)
     -> datadog.trace.id: Str(17149159500458815648)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> http.status_code: Str(200)
     -> language: Str(jvm)
     -> component: Str(tomcat-server)
     -> deployment.environment: Str(production)
     -> peer.ipv4: Str(127.0.0.1)
     -> span.kind: Str(server)
     -> http.useragent: Str(Java-http-client/17.0.13)
     -> _dd.tracer_host: Str(cyrille-le-clerc-macbook.local)
     -> servlet.path: Str(/shipOrder)
     -> http.method: Str(POST)
     -> servlet.context: Str(/)
     -> http.route: Str(/shipOrder)
     -> runtime-id: Str(9136fc59-3ff8-4bf1-8ec6-725b8c6b8ef3)
     -> _dd.p.dm: Str(-0)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> http.url: Str(http://shipping.local:8088/shipOrder)
     -> service.version: Str(1.1)
     -> http.hostname: Str(shipping.local)
     -> _dd.top_level: Double(1)
     -> thread.id: Double(38)
     -> process.pid: Double(43095)
     -> _dd.profiling.enabled: Double(0)
     -> peer.port: Double(59969)
     -> _dd.trace_span_attribute_schema: Double(0)
     -> _sampling_priority_v1: Double(1)
     -> _dd.measured: Double(1)
  • Adopt new OTel Semantic Conventions http.request.method and http.response.status_code instead of http.method and http.status_code
  • The span name should be ${http.request.method} ${http.route}, the attribute dd.span.Resource is right for this span type
    • OTTL transformation to fix the pb: set (name, attributes["dd.span.Resource"]) where name == "servlet.request"

Misalignment applying to internal Spring Framework spans(span name spring.handler)

This probably applies to other internal spans.

Example Datadog internal span produced by a Spring Boot app (code) and converted by the OTel Col Datadog receiver

Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> service.version: Str(1.1)
     -> telemetry.sdk.version: Str(Datadog-1.44.1~13a9a2d011)
     -> telemetry.sdk.name: Str(Datadog)
     -> telemetry.sdk.language: Str(java)
     -> process.runtime.version: Str(17.0.13)
     -> service.name: Str(shipping)
     -> host.name: Str(localhost)
     -> os.type: Str(darwin)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope Datadog 1.44.1~13a9a2d011
Span #1
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 3e65de74bd03a50a
    ID             : 6a0035efa46106ae
    Name           : spring.handler
    Kind           : Server
    Start time     : 2024-12-23 10:28:03.720530834 +0000 UTC
    End time       : 2024-12-23 10:28:03.777623 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(ShippingController.shipOrder)
     -> datadog.span.id: Str(7638164271391639214)
     -> datadog.trace.id: Str(17149159500458815648)
     -> service.version: Str(1.1)
     -> span.kind: Str(server)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> language: Str(jvm)
     -> component: Str(spring-web-controller)
     -> deployment.environment: Str(production)
     -> _dd.measured: Double(1)
     -> thread.id: Double(38)
  • Span should be kind = SpanKind.SPAN_KIND_INTERNAL
    • OTTL transformation to fix the pb: set (kind, SPAN_KIND_INTERNAL) where kind == SPAN_KIND_SERVER and name == "spring.handler"
  • Span name should be the Java method nam hat is carried over by the dd.span.Resource attribute
    • OTTL transformation to fix the pb: `set (name, attributes["dd.span.Resource"]) where name

Misalignment applying to Database client spans

Specs: https://opentelemetry.io/docs/specs/semconv/database/database-spans/

Example Datadog DB client span produced by a Spring Boot app (code and converted by the OTel Col Datadog receiver

ResourceSpans #1
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> service.version: Str(1.1)
     -> telemetry.sdk.version: Str(Datadog-1.44.1~13a9a2d011)
     -> telemetry.sdk.name: Str(Datadog)
     -> telemetry.sdk.language: Str(java)
     -> process.runtime.version: Str(17.0.13)
     -> service.name: Str(postgresql)
     -> host.name: Str(localhost)
     -> os.type: Str(darwin)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope Datadog 1.44.1~13a9a2d011
Span #0
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 6a0035efa46106ae
    ID             : 2dc4a71b6d1cac05
    Name           : postgresql.query
    Kind           : Client
    Start time     : 2024-12-23 10:28:03.723354292 +0000 UTC
    End time       : 2024-12-23 10:28:03.775606 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(select pg_sleep(?))
     -> datadog.span.id: Str(3297944563378596869)
     -> datadog.trace.id: Str(17149159500458815648)
     -> db.type: Str(postgresql)
     -> db.user: Str(my_shopping_cart)
     -> deployment.environment: Str(production)
     -> component: Str(java-jdbc-statement)
     -> span.kind: Str(client)
     -> _dd.base_service: Str(shipping)
     -> db.operation: Str(select)
     -> peer.hostname: Str(postgresql.local)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> db.instance: Str(my_shopping_cart)
     -> db.pool.name: Str(HikariPool-1)
     -> _dd.measured: Double(1)
     -> _dd.top_level: Double(1)
     -> thread.id: Double(38)
Span #1
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 6a0035efa46106ae
    ID             : 504c8390c5d6dc9b
    Name           : postgresql.query
    Kind           : Client
    Start time     : 2024-12-23 10:28:03.72148425 +0000 UTC
    End time       : 2024-12-23 10:28:03.723227917 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(postgresql.query)
     -> sampling.priority: Str(1.000000)
     -> datadog.span.id: Str(5786144279081966747)
     -> datadog.trace.id: Str(17149159500458815648)
     -> span.kind: Str(client)
     -> _dd.base_service: Str(shipping)
     -> deployment.environment: Str(production)
     -> component: Str(java-jdbc-statement)
     -> db.type: Str(postgresql)
     -> db.instance: Str(my_shopping_cart)
     -> db.user: Str(my_shopping_cart)
     -> peer.hostname: Str(postgresql.local)
     -> db.pool.name: Str(HikariPool-1)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> _sampling_priority_v1: Double(1)
     -> _dd.measured: Double(1)
     -> _dd.top_level: Double(1)
     -> thread.id: Double(38)
  • The resource attribute service.name should be set to the value of the span attribute _dd.base_service
  • Rename the following span attributes
    • db.type -> db.system
    • db.operation -> db.operation.name
    • db.instance -> db.name
  • The span name should be set to ${db.operation.name} ${}

Steps to Reproduce

  • Instrument a Spring Boot App similar to this example with the Datadog Java Agent v1.44.1, send the traces to and OTel Col Datadog Receiver using the DD_TRACE_AGENT_URL="http://localhost:8126", note that setting DD_REMOTE_CONFIGURATION_ENABLED=false seems to prevent some problems of traces not being exported:

    java \
     -javaagent:"$DATADOG_AGENT_JAR" \
     -Dserver.port=8088 \
     -jar target/shipping-1.1-SNAPSHOT.jar

  • Setup an OTel Collector v0.116.0 with a Datadog Receiver and exporting traces to the console for debug

    receivers:
  datadog:
    endpoint: localhost:8126
    read_timeout: 60s
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [datadog]
      processors: 
      exporters: [debug]

Expected Result

OTel-compliant spans reported in the Otel Col stdout

Actual Result

The non OTel-compliant spans described above

Collector version

v0.116.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  datadog:
    endpoint: localhost:8126
    read_timeout: 60s
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [datadog]
      processors: 
      exporters: [debug]

Log output

see issue description above

Additional context

No response
@cyrille-leclerc cyrille-leclerc added bug Something isn't working needs triage New item requiring triage labels Dec 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot mentioned this issue Dec 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage New item requiring triage receiver/datadog
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cyrille-leclerc