Resharding #
Syntax | Description | Type |
---|---|---|
ALTER RESHARDING RULE | Modify resharding rule | RAL |
RESHARD TABLE tableName by rule | Start resharding | RDL |
SHOW RESHARDING RULE | View resharding rule | RAL |
SHOW RESHARDING LIST | View resharding job list | RAL |
SHOW RESHARDING STATUS jobId | View resharding job status | RAL |
START RESHARDING jobId | Start stopped resharding job | RAL |
STOP RESHARDING jobId | Stop resharding job | RAL |
SHOW RESHARDING CHECK ALGORITHMS | View consistency check algorithm | RAL |
CHECK RESHARDING jobId | Execute resharding data consistency check | RAL |
START RESHARDING CHECK jobId | Start stopped resharding data consistency check job | RAL |
STOP RESHARDING CHECK jobId | Stopped resharding data consistency check job | RAL |
DROP RESHARDING CHECK jobId | Delete consistency check job | RAL |
COMMIT RESHARDING jobId | Commit resharding job | RAL |
FORCE COMMIT RESHARDING jobId | Force commit resharding job | RAL |
ROLLBACK RESHARDING jobId | Rollback resharding job | RAL |
APPLY RESHARDING jobId | Apply new resharding configurations (toggle metadata) | RAL |
FORCE APPLY RESHARDING jobId | Force apply new resharding configurations (toggle metadata) | RAL |
STOP RESHARDING SOURCE WRITING jobId | Stop writing | RAL |
RESTORE RESHARDING SOURCE WRITING jobId | Restore stop writing | RAL |
AUTO RESHARD TABLE tableName BY rule | Resharding auto mode | RDL |
RESHARD SINGLE TABLE tableName TO BROADCAST | Expand a single table into a broadcast table | RDL |
AUTO RESHARD SINGLE TABLE tableName TO BROADCAST | Single table expansion into broadcast table automatic mode | RDL |
1. Modify Resharding Rule #
ALTER RESHARDING RULE
Example
ALTER RESHARDING RULE (READ(RATE_LIMITER (TYPE(NAME='QPS’,PROPERTIES(‘qps'='5000’)))))
Config Description
ALTER RESHARDING RULE (
READ( -- Data reading configuration. If it is not configured, part of the parameters will take effect by default.
WORKER_THREAD=20, -- Obtain the thread pool size of all the data from the source side. If it is not configured, the default value is used.
BATCH_SIZE=1000, -- The maximum number of records returned by a query operation. If it is not configured, the default value is used.
SHARDING_SIZE=10000000, -- Sharding size of all the data. If it is not configured, the default value is used.
RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
TYPE( -- Algorithm type. Option: QPS
NAME='QPS',
PROPERTIES( -- Algorithm property
'qps'='500'
)))
),
WRITE( -- Data writing configuration. If it is not configured, part of the parameters will take effect by default.
WORKER_THREAD=20, -- The size of the thread pool on which data is written into the target side. If it is not configured, the default value is used.
BATCH_SIZE=1000, -- The maximum number of records for a batch write operation. If it is not configured, the default value is used.
RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
TYPE( -- Algorithm type. Option: TPS
NAME='TPS',
PROPERTIES( -- Algorithm property.
'tps'='2000'
)))
),
STREAM_CHANNEL ( -- Data channel. It connects producers and consumers, used for reading and writing procedures. If it is not configured, the MEMORY type is used by default.
TYPE( -- Algorithm type. Option: MEMORY
NAME='MEMORY',
PROPERTIES( -- Algorithm property
'block-queue-size'='2000' -- Property: blocking queue size.
)))
);
2. Start Resharding #
RESHARD TABLE tableName BY rule
Parameter Description
Name | Description |
---|---|
tableName | Resharding table |
Example
RESHARD TABLE t_order BY(
STORAGE_UNITS(ds_0, ds_1, ds_2),
SHARDING_COLUMN=order_id,
TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);
3. View Resharding Rule #
SHOW RESHARDING LIST;
Parameter Description
None
Example
SHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id | tables | sharding_total_count | active | create_time | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2 | true | 2022-10-25 10:10:58 | NULL |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
4. View Resharding Job List #
SHOW RESHARDING LIST;
Parameter Description
None
Example
SHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id | tables | sharding_total_count | active | create_time | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2 | true | 2022-10-25 10:10:58 | NULL |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
5. View Resharding Job Status #
SHOW RESHARDING STATUS jobId;
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
SHOW RESHARDING STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| item | data_source | status | active | processed_records_count | inventory_finished_percentage | remaining_seconds | incremental_idle_seconds | error_message |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| 0 | ds_0 | EXECUTE_INCREMENTAL_TASK | true | 3 | 100 | 0 | 92 | |
| 1 | ds_1 | EXECUTE_INCREMENTAL_TASK | true | 3 | 100 | 0 | 92 | |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
6. Start Stopped Resharding Job #
START RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
START RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
7. Stop Resharding Job #
STOP RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
STOP RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
8. View data consistency check algorithm #
For consistency check algorithm selection, please refer to Data Consistency Check Algorithm。
SHOW RESHARDING CHECK ALGORITHMS
Example
SHOW RESHARDING CHECK ALGORITHMS;
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| type | type_aliases | supported_database_types | description |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| CRC32_MATCH | | MySQL,MariaDB,H2,Aurora | Match CRC32 of records. |
| DATA_MATCH | | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Match raw data of records. |
| SphereEx:BATCH_CRC32_MATCH | SphereEx:DATA_CONSISTENCY_CHECK_CRC32_MATCH | MySQL,MariaDB,H2,Aurora | Match CRC32 of records (Commercial edition). |
| SphereEx:DATA_DIFF | | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Diff raw data of records. |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
Output Description
- type: algorithm name
- type_aliases: algorithm aliases
- supported_database_types: Which databases the algorithm supports
- description: Algorithm description
9. Check Resharding Data Consistency #
CHECK RESHARDING jobId BY TYPE (NAME='algorithmType');
Parameter Description
Name | Description |
---|---|
jobid | Job id |
algorithmType | Consistency check algorithm type |
Example
Specify the jobId to use specified verification algorithm to perform data consistency verification
CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='CRC32_MATCH');
CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='DATA_MATCH', PROPERTIES('chunk-size'='1000'));
CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:BATCH_CRC32_MATCH');
CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:DATA_DIFF', PROPERTIES('diff-storage-type'='DATABASE','chunk-size'='1000','diff-storage-unit-name'='ds_2','diff-table-name'='spex_cnschk_1','incremental-idle-seconds-threshold'='30'));
10. Start Stopped Resharding Data Consistency Check Job #
START RESHARDING CHECK jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
START RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';
11. Stop Resharding Data Consistency Check Job #
STOP RESHARDING CHECK jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
STOP RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';
12. Delete Resharding Data Consistency Check Job #
DROP RESHARDING CHECK jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
DROP RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';
13. Commit Resharding Job #
COMMIT RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
14. Force Commit Resharding Job #
FORCE COMMIT RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
FORCE COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
15. Rollback Resharding Job #
ROLLBACK RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
ROLLBACK RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
16. Apply New Resharding Configurations (toggle metadata) #
APPLY RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
17. Force Apply New Resharding Configurations (toggle metadata) #
FORCE APPLY RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
FORCE APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
18. Stop Writing #
STOP RESHARDING SOURCE WRITING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
STOP RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';
19. Restore Stop Writing #
RESTORE RESHARDING SOURCE WRITING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
RESTORE RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';
20. Resharding Auto Mode #
AUTO RESHARD TABLE tableName BY rule
Parameter Description
Name | Description |
---|---|
tableName | Resharding table |
Example
AUTO RESHARD TABLE t_order BY(
STORAGE_UNITS(ds_0, ds_1, ds_2),
SHARDING_COLUMN=order_id,
TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);
21. Expand a single table into a broadcast table #
RESHARD SINGLE TABLE tableName TO BROADCAST
Parameter Description
Name | Description |
---|---|
tableName | Resharding single table |
Example
RESHARD SINGLE TABLE t_address TO BROADCAST
22. Single table expansion into broadcast table automatic mode #
AUTO RESHARD SINGLE TABLE tableName TO BROADCAST
Parameter Description
Name | Description |
---|---|
tableName | Resharding single table |
Example
AUTO RESHARD SINGLE TABLE t_address TO BROADCAST