EOFError when connecting with SCRAM authentication

[sandbardev]

Hello! We've been trying to use ruby-kafka to connect to one of our brokers but we can't seem to get it to work.

• Version of Ruby: 2.7.5
• Version of Kafka: 2.7.1
• Version of ruby-kafka: 1.4

Steps to reproduce

require 'kafka'

@kafka = Kafka.new(
  ['url_1:9096', 'url_2:9096'],
  sasl_over_ssl: false,
  sasl_scram_username: 'valid_user',
  sasl_scram_password: 'valid_password',
  sasl_scram_mechanism: 'sha512',
  ssl_verify_hostname: false,
  client_id: 'app_name'
)

@kafka.topics

We immediately get the error:

Traceback (most recent call last):
	7: from connect.rb:13:in `<main>'
	6: from /home/m4u-sanderson/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.4.0/lib/kafka/client.rb:706:in `topics'
	5: from /home/m4u-sanderson/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.4.0/lib/kafka/cluster.rb:385:in `list_topics'
	4: from /home/m4u-sanderson/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.4.0/lib/kafka/cluster.rb:458:in `random_broker'
	3: from /home/m4u-sanderson/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.4.0/lib/kafka/cluster.rb:109:in `refresh_metadata_if_necessary!'
	2: from /home/m4u-sanderson/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.4.0/lib/kafka/cluster.rb:105:in `refresh_metadata!'
	1: from /home/m4u-sanderson/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.4.0/lib/kafka/cluster.rb:405:in `cluster_info'
/home/m4u-sanderson/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/gems/ruby-kafka-1.4.0/lib/kafka/cluster.rb:454:in `fetch_cluster_info': Could not connect to any of the seed brokers: (Kafka::ConnectionError)
- kafka://url_1:9096: Connection error EOFError: EOFError
- kafka://url_2:9096: Connection error EOFError: EOFError

However, we manage to succesfulu connect to the same broker with the same credentials using the following python code:

from kafka import KafkaAdminClient

client = KafkaAdminClient(
    bootstrap_servers='url_1:9096,url_2:9096',
    security_protocol='SASL_SSL',
    sasl_mechanism='SCRAM-SHA-512',
    ssl_check_hostname=False,
    sasl_plain_username='valid_user',
    sasl_plain_password='valid_password',
)

topics = client.list_topics()

We tried downgrading ruby-kafka to v0.5 as shown by the support tables in the README, but got the exact same error (We understand that Kafka 2.7.1 might not be compatible).

Can we get any help on this?

[dasch]

This is not just due to sasl_over_ssl needing to be true?

[sandbardev]

Not sure. sasl_ver_ssl: true forced us to provide a pair of credentials which we didn't have, and couldn't create a pair recognized by the Kafka servers.
What solved the issue for us was adding ssl_ca_certs_from_system: true:

@kafka = Kafka.new(
      ['url_1:9096', 'url_2:9096'],
      ssl_ca_certs_from_system: true,
      ssl_verify_hostname: false,
      sasl_scram_username: 'valid_user',
      sasl_scram_password: 'valid_password',
      sasl_scram_mechanism: 'sha512',
      client_id: 'app_name'
)

It then worked perfectly. We can close this issue.