Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mysql 实时同步 msyql 出错 #8342

Open
2 of 3 tasks
sandea opened this issue Dec 19, 2024 · 6 comments
Open
2 of 3 tasks

mysql 实时同步 msyql 出错 #8342

sandea opened this issue Dec 19, 2024 · 6 comments
Labels
bug

Comments

@sandea
Copy link

sandea commented Dec 19, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

mysql 实时同步 msyql 出错

SeaTunnel Version

2.3.8

SeaTunnel Config

env {
  # You can set SeaTunnel environment configuration here
  execution.parallelism = 2
  job.mode = "STREAMING"
  # 10秒检查一次，可以适当加大这个值
  checkpoint.interval = 10000
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

# 配置数据源
source {
  MySQL-CDC {
    # 数据库账号
    username = "root"
    password = "123456"
    # 源表，格式：数据库名.表名
    table-names = ["source.test_table"]
    base-url = "jdbc:mysql://127.0.0.1:3306/source?useSSL=false&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=true"
    startup.mode = "initial"
  }
}
# 配置目标库
sink {

  jdbc {
    url = "jdbc:mysql://127.0.0.1:3306/sink?useSSL=false&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=true"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "root"
    password = "123456"

    generate_sink_sql = true
        # 目标数据库名
    database = "sink"
    # 目标表名
    table = "test_table"
    # 主键名称
    primary_keys = ["id"]
    field_ide = LOWERCASE
    schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
    data_save_mode="APPEND_DATA"
  }

}

Running Command

./bin/seatunnel.sh --config config/mysql_to_mysql_config -m local

Error Exception

Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:172)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:862)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:444)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:230)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:226)
	at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:244)
	at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.config.CustomMySqlConnectionConfiguration$1.connect(CustomMySqlConnectionConfiguration.java:52)
	at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:888)
	at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:883)
	at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:548)
	at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:496)
	at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.utils.TableDiscoveryUtils.listTables(TableDiscoveryUtils.java:47)
	at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.discoverDataCollections(MySqlDialect.java:97)
	... 9 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:59)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:103)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:149)
	at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:165)
	at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:355)
	at com.mysql.cj.protocol.a.NativeAuthenticationProvider.negotiateSSLConnection(NativeAuthenticationProvider.java:789)
	at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.java:499)
	at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:217)
	at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1411)
	at com.mysql.cj.NativeSession.connect(NativeSession.java:165)
	at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:982)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:852)
	... 20 more
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
	at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)
	at sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:103)
	at sun.security.ssl.TransportContext.kickstart(TransportContext.java:227)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:433)
	at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:213)
	at com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:206)
	at com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:99)
	at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:350)
	... 27 more

	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:390) [seatunnel-starter.jar:2.3.8]
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:184) [seatunnel-starter.jar:2.3.8]
	at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48) [seatunnel-starter.jar:2.3.8]
	at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:44) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.8]
2024-12-19 16:18:58,772 INFO  [.s.e.s.c.CheckpointCoordinator] [hz.main.generic-operation.thread-38] - start clean pending checkpoint cause CheckpointCoordinator inside have error.
2024-12-19 16:18:58,773 INFO  [.s.e.s.c.CheckpointCoordinator] [hz.main.generic-operation.thread-38] - Turn checkpoint_state_922040277638578177_1 state from RUNNING to FAILED
2024-12-19 16:18:58,773 WARN  [o.a.s.e.s.d.p.SubPlan         ] [hz.main.generic-operation.thread-38] - Job SeaTunnel_Job (922040277638578177), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline
2024-12-19 16:18:58,778 INFO  [o.a.s.e.s.d.p.SubPlan         ] [hz.main.generic-operation.thread-38] - Job SeaTunnel_Job (922040277638578177), Pipeline: [(1/1)] turned from state RUNNING to CANCELING.
2024-12-19 16:18:58,778 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.generic-operation.thread-38] - Job SeaTunnel_Job (922040277638578177), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] state process is start
2024-12-19 16:18:58,779 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.generic-operation.thread-38] - Job SeaTunnel_Job (922040277638578177), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] turned from state RUNNING to CANCELING.
2024-12-19 16:18:58,779 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.generic-operation.thread-38] - Send cancel Job SeaTunnel_Job (922040277638578177), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] operator to member [localhost]:5802
2024-12-19 16:18:58,779 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.generic-operation.thread-38] - [localhost]:5802 [seatunnel-443671] [5.1] Task (TaskGroupLocation{jobId=922040277638578177, pipelineId=1, taskGroupId=1}) need cancel.
2024-12-19 16:18:58,780 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.generic-operation.thread-38] - Job SeaTunnel_Job (922040277638578177), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SourceTask (1/1)] state process is start
2024-12-19 16:18:58,780 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=922040277638578177, pipelineId=1, taskGroupId=1}] - [localhost]:5802 [seatunnel-443671] [5.1] Interrupted task 20000 - org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask@470ae3c
2024-12-19 16:18:58,780 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=922040277638578177, pipelineId=1, taskGroupId=1}] - [localhost]:5802 [seatunnel-443671] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=922040277638578177, pipelineId=1, taskGroupId=1}
2024-12-19 16:18:58,783 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=922040277638578177, pipelineId=1, taskGroupId=1}] - [localhost]:5802 [seatunnel-443671] [5.1] taskGroup TaskGroupLocation{jobId=922040277638578177, pipelineId=1, taskGroupId=1} complete with CANCELED
2024-12-19 16:18:58,783 ERROR [.s.e.s.c.CheckpointCoordinator] [hz.main.generic-operation.thread-39] - report error from task
org.apache.seatunnel.common.utils.SeaTunnelException: java.util.concurrent.CancellationException
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
	at org.apache.seatunnel.engine.server.task.AbstractTask.close(AbstractTask.java:102)
	at org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask.close(SourceSplitEnumeratorTask.java:123)
	at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:710)
	at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018)
	at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

	at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:390) [seatunnel-starter.jar:2.3.8]
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:184) [seatunnel-starter.jar:2.3.8]
	at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48) [seatunnel-starter.jar:2.3.8]
	at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:44) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123) [seatunnel-starter.jar:2.3.8]
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [seatunnel-starter.jar:2.3.8]

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

No response

Screenshots

image
image
image

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@sandea sandea added the bug label Dec 19, 2024
@FrommyMind
Copy link
Contributor

Which version of mysql do you using？
And can you check your mysql variables with :

show variables like 'tls_version';

@sandea
Copy link
Author

sandea commented Dec 19, 2024

mysql 5.7

@sandea
Copy link
Author

sandea commented Dec 19, 2024

image

@sandea
Copy link
Author

sandea commented Dec 19, 2024

Which version of mysql do you using？ And can you check your mysql variables with :

show variables like 'tls_version';

image

@sandea
Copy link
Author

sandea commented Dec 19, 2024

batch data is success，stream mode is report there errors

@FrommyMind
Copy link
Contributor

Add enabledTLSProtocols=TLSv1.2 to you jdbc url and retry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@FrommyMind @sandea