[exporter/elasticsearch] Ability to specify the document ID for logs

[mauri870]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

In Beats we are using the bodymap mapping mode to control the final document structure for logs. There is one thing we are unable to control with this approach, the final document ID. We use to control this ID for means of deduplicating messages. For context, see https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-deduplication.html.

Describe the solution you'd like

We would like to have a way to tell the exporter to use a specific ID as the document identifier. The current implementation does not allow this and defaults to an autogenerated ID here. This might be a broader question for the collector in general, as we would like such functionality to be available from all the exporters.

Describe alternatives you've considered

One alternative is to consider a specific field in the log message body as the document id. For example in beats we use @metatada._id as the ID field, this @metadata field is then stripped out at our output layer (equivalent to an exporter in OTel) so we can use this information to set the ID properly in the final document.

Looking at the Logs model for OTel this would be equivalent to an Attribute. I imagine deduplication is an interesting functionality that perhaps shoudn't be constrained to the bodymap mapping mode or the elasticsearchexporter so I'm open to more general suggestions.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

This sounds like a valid enhancement to me.
We can probably have something like log_dynamic_index, to allow setting _id from attributes.

Let's see what the code owners have to say about this.