[otel-data] Add @timestamp as sort field for logs and traces #114049
Conversation
gregkalapos
added
>non-issue
:Data Management/Data streams
|
Pinging @elastic/es-data-management (Team:Data Management) |
elasticsearchmachine
added
Team:Data Management
| @@ -8,7 +8,7 @@ template: | |||
| index: | |||
| mode: logsdb | |||
| sort: | |||
| field: [ "resource.attributes.host.name" ] | |||
| field: [ "@timestamp", "resource.attributes.host.name" ] | |||
There was a problem hiding this comment.
Do we still need resource.attributes.host.name? Why was it there in the first place?
There was a problem hiding this comment.
From what i understand the default for this setting is @timestamp and host.name. I took this from @felixbarny in #113397
So our goal is to just mimic the default.
I think alias passthrough does not help here, so we essentially just mimic the default, but in the otel case host.name is resource.attributes.host.name.
So I think we still want it there.
There was a problem hiding this comment.
The default sorts first by host name, then by timestamp. That’s an important detail that we should keep. The result is that the metadata for that host can be deduplicated because there are a a lot of the same values next to each other.
If we sort by timestamp first, only documents with the exact same timestamp are sorted by host name.
gregkalapos
force-pushed
the
add_timestamp_as_sort_field
branch
from
1050b9b
to
3634c4b
Compare
| @@ -70,3 +70,24 @@ setup: | |||
| - match: { hits.hits.0.fields.error\.exception\.type: ["MyException"] } | |||
| - match: { hits.hits.0.fields.error\.exception\.message: ["foo"] } | |||
| - match: { hits.hits.0.fields.error\.stack_trace: ["Exception in thread \"main\" java.lang.RuntimeException: Test exception\n at com.example.GenerateTrace.methodB(GenerateTrace.java:13)\n at com.example.GenerateTrace.methodA(GenerateTrace.java:9)\n at com.example.GenerateTrace.main(GenerateTrace.java:5)"] } | |||
| --- | |||
| "@timestamp should be used as sort field": | |||
There was a problem hiding this comment.
I'm not sure about these tests. It feels like they may rely too much on an implementation details that _search returns documents in order of how they're indexed. I don't think there's a guarantee on the ordering of search results when no sorting or scoring is involved. We're not sorting because we want the search results to return in a specific order but because it has benefits wrt storage savings. A unit tests isn't the best way to verify that. So maybe just check that the index setting has been applied or even just remove the tests.
Solves #113397.