Kafka connect without magic byte serialization

[woile]

Hi there, I hope this is the right place to ask this, I'm having trouble finding the correct piece of documentation in the ecosystem.

I'm using a custom serialization format in kafka where the schema is in the headers, not in the magic byte, like as specified by confluent.

If I understand apicurio correctly, I should be able to create a serde that reads the schema from the headers, right?
Would it also be possible to use this serde with kafka-connect? Is there any example or guide on how to do this?

Thanks in advance 🙏🏻

[EricWittmann]

Not only can you create custom serde classes, you may be able to just use ours. We have all sorts of config options for our provided serializers and deserializers. One of the options you can configure is where to find the globalId (or alternatively the GAV information). Docs for configuring our SerDes can be found here:

https://www.apicur.io/registry/docs/apicurio-registry/2.6.x/getting-started/assembly-configuring-kafka-client-serdes.html

Note that it never reads the schema from the headers, but rather it reads the "coordinates" of the schema in the registry. The coordinates can either be the globalId or the combination of groupId, artifactId, and version.

[carlesarnal]

@woile apart from what Eric said, if you want to take a look at an example we have a repository with a complete example using Kafka, Kafka connect etc where one connector is taking info from a postgresql database and syncing some elasticsearch indexes with fully configured connectors. You have the postgresql connector you can use as an example here. This is possible thanks to our converters, that are already included in Debezium in case you're using it. If not, you must build your own connect distro including our converters.