Skip to content

Commit

Permalink
[Doc] Add documentation for 1.2.9
Browse files Browse the repository at this point in the history 
Signed-off-by: PengFei Li <[email protected]>
  • Loading branch information
@banmoy
banmoy committed Dec 14, 2023
1 parent bd9e6bd commit bfd339f
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 1 deletion.
18 changes: 17 additions & 1 deletion docs/content/connector-sink.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The Flink connector supports DataStream API, Table API & SQL, and Python API. It

| Connector | Flink | StarRocks | Java | Scala |
|-----------|--------------------------|---------------| ---- |-----------|
| 1.2.9 | 1.15,1.16,1.17,1.18 | 2.1 and later| 8 | 2.11,2.12 |
| 1.2.8 | 1.13,1.14,1.15,1.16,1.17 | 2.1 and later| 8 | 2.11,2.12 |
| 1.2.7 | 1.11,1.12,1.13,1.14,1.15 | 2.1 and later| 8 | 2.11,2.12 |

Expand Down Expand Up @@ -100,7 +101,7 @@ In your Maven project's `pom.xml` file, add the Flink connector as a dependency
| sink.buffer-flush.max-rows | No | 500000 | The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. This parameter is available only when `sink.version` is `V1` and `sink.semantic` is `at-least-once`. Valid values: 64000 to 5000000. |
| sink.buffer-flush.interval-ms | No | 300000 | The interval at which data is flushed. This parameter is available only when `sink.semantic` is `at-least-once`. Valid values: 1000 to 3600000. Unit: ms. |
| sink.max-retries | No | 3 | The number of times that the system retries to perform the Stream Load job. This parameter is available only when you set `sink.version` to `V1`. Valid values: 0 to 10. |
| sink.connect.timeout-ms | No | 1000 | The timeout for establishing HTTP connection. Valid values: 100 to 60000. Unit: ms. |
| sink.connect.timeout-ms | No | 30000 | The timeout for establishing HTTP connection. Valid values: 100 to 60000. Unit: ms. Before 1.2.9, the default value is 1000. |
| sink.wait-for-continue.timeout-ms | No | 10000 | Supported since 1.2.7. The timeout for waiting response of HTTP 100-continue from the FE. Valid values: `3000` to `600000`. Unit: ms |
| sink.ignore.update-before | No | true | Supported since version 1.2.8. Whether to ignore `UPDATE_BEFORE` records from Flink when loading data to Primary Key tables. If this parameter is set to false, the record is treated as a delete operation to StarRocks table. |
| sink.parallelism | No | NONE | The parallelism of loading. Only available for Flink SQL. If this parameter is not specified, Flink planner decides the parallelism. **In the scenario of multi-parallelism, users need to guarantee data is written in the correct order.** |
Expand Down Expand Up @@ -214,6 +215,21 @@ The Flink connector provides the following metrics to monitor loading.
| totalFlushFailedTimes | counter | number of times that the flushing been failed. |
| totalFilteredRows | counter | number of rows filtered which is also included in totalFlushRows. |

### Flink CDC Synchronization

You can use [Flink CDC 3.0](https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html) framework
to easily build a streaming ELT pipeline from CDC sources (such as MySQL, Kafka) to StarRocks. The pipeline can synchronize whole
database, merged sharding tables, and schema changes.

The flink connector has been integrated into the framework as [StarRocks Pipeline Connector](https://ververica.github.io/flink-cdc-connectors/master/content/pipelines/starrocks-pipeline.html).
The pipeline connector supports
* create database/table automatically
* schema change synchronization
* full and incremental data synchronization

You can see [Streaming ELT from MySQL to StarRocks using Flink CDC 3.0](https://ververica.github.io/flink-cdc-connectors/master/content/quickstart/mysql-starrocks-pipeline-tutorial.html)
for a quick start.

## Examples

The following examples show how to use the Flink connector to load data into a StarRocks table with Flink SQL or Flink DataStream.
Expand Down
1 change: 1 addition & 0 deletions docs/content/connector-source.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Unlike the JDBC connector provided by Flink, the Flink connector of StarRocks su

| Connector | Flink | StarRocks | Java | Scala |
|-----------|--------------------------|---------------| ---- |-----------|
| 1.2.9 | 1.15,1.16,1.17,1.18 | 2.1 and later| 8 | 2.11,2.12 |
| 1.2.8 | 1.13,1.14,1.15,1.16,1.17 | 2.1 and later| 8 | 2.11,2.12 |
| 1.2.7 | 1.11,1.12,1.13,1.14,1.15 | 2.1 and later| 8 | 2.11,2.12 |

Expand Down

0 comments on commit bfd339f

Please sign in to comment.