[Fix][Plugin]Add remote plugin discovery for flink yarn applicion deploy mode

[hanhanzhang]

Solve the issue: #7321

[corgy-w]

Hi,Please follow guide to open ci on your fork repository in github.
this

[Hisoka-X]

Thanks @hanhanzhang ! Could you add a test case for this?

[Hisoka-X]

cc @TyrantLucifer

[Carl-Zhou-CN]

Restore it

[TyrantLucifer]

in my opinion, we natively support application mode, but we need to change the packaging mode

[hanhanzhang]

in my opinion, we natively support application mode, but we need to change the packaging mode

yes，we can use connector-plugins as a yarn-ship files parameter. During flink deployment, connector plugins is uploaded to an external storage system. If there are many plugins, task deployment slows down.

[hanhanzhang]

Remote plugin discover mechanism allows user to upload connector plugins to an external storage system in advance. In application deployment mode, dependency plugin can be loaded from an external storage system as required.

[Carl-Zhou-CN]

${FLINK_HOME}/bin/flink run-application -Dyarn.ship-files='./config/v2.streaming.conf.template' -Dyarn.classpath.include-user-jar=DISABLED -Dyarn.provided.usrlib.dir='hdfs://HDFS4001185/user/flink/usrlib/' --target yarn-application -c org.apache.seatunnel.core.starter.flink.SeaTunnelFlink /usr/local/service/seatunnel/starter/seatunnel-flink-15-starter.jar --config v2.streaming.conf.template --name SeaTunnel

Using the combination of yarn.provided.usrlib.dir along with yarn.ship-files seems to be a good approach.

[Carl-Zhou-CN]

[hanhanzhang]

${FLINK_HOME}/bin/flink run-application -Dyarn.ship-files='./config/v2.streaming.conf.template' -Dyarn.classpath.include-user-jar=DISABLED -Dyarn.provided.usrlib.dir='hdfs://HDFS4001185/user/flink/usrlib/' --target yarn-application -c org.apache.seatunnel.core.starter.flink.SeaTunnelFlink /usr/local/service/seatunnel/starter/seatunnel-flink-15-starter.jar --config v2.streaming.conf.template --name SeaTunnel

Using the combination of yarn.provided.usrlib.dir along with yarn.ship-files seems to be a good approach.

Yes, I've thought about it this way. I wouldn't recommend putting SeaTunnel lib in yarn.provided.lib.dirs. This would require SeaTunnel and Flink dependency packages to be placed together. I think it might be a good way to use SeaTunnel Connector as a UDF similar to SQL.

In addition, the yarn.provided.usrlib.dirs means all the jars in the classpath, the initial plugin design should be to put the plugins needed for the task in the classpath. What do you think?

[Carl-Zhou-CN]

yarn.provided.usrlib.dir

I am very sorry for my oversight, -Dyarn.provided.usrlib.dir is supported starting from Flink 1.16.

[Hisoka-X]

Please share some details about how to use this new feature?

[Carl-Zhou-CN]

command ${SEATUNNEL_HOME}/bin/start-seatunnel-flink-15-connector-v2.sh --config /usr/seatunnel/temp/test.conf --deploy-mode run-application --connectors "hdfs://localhost:8020/seatunnel/2.3.7/connectors" --target yarn-application --yarnqueue default --yarnjobManagerMemory 1024 --yarntaskManagerMemory 1024

[TyrantLucifer]

bin/start-seatunnel-flink-15-connector-v2.sh --config config/v2.batch.config.template --target yarn-application -e run-application --name st-test -Dyarn.application.name=st-test -Dyarn.ship-files="config/v2.batch.config.template\;connectors"

this command can support application mode submit, the act of directly pulling jar packages from HDFS should be executed by Flink, and I think encapsulating them in the client of SeaTunnel is a bit too heavy. Is there any other solution that can directly take advantage of Flink's capabilities?

[hanhanzhang]

bin/start-seatunnel-flink-15-connector-v2.sh --config config/v2.batch.config.template --target yarn-application -e run-application --name st-test -Dyarn.application.name=st-test -Dyarn.ship-files="config/v2.batch.config.template\;connectors"

this command can support application mode submit, the act of directly pulling jar packages from HDFS should be executed by Flink, and I think encapsulating them in the client of SeaTunnel is a bit too heavy. Is there any other solution that can directly take advantage of Flink's capabilities?

The design allows users to pre-upload Plugin to external storage systems, such as hdfs, s3, etc. flink application deployment only needs to guide the plugin to the external storage system address.