Resharding #
Syntax | Description | Type |
---|---|---|
ALTER RESHARDING RULE | Modify resharding rule | RAL |
RESHARD TABLE tableName by rule | Start resharding | RDL |
SHOW RESHARDING RULE | View resharding rule | RAL |
SHOW RESHARDING LIST | View resharding job list | RAL |
SHOW RESHARDING STATUS jobId | View resharding job status | RAL |
START RESHARDING jobId | Start stopped resharding job | RAL |
STOP RESHARDING jobId | Stop resharding job | RAL |
SHOW RESHARDING CHECK ALGORITHMS | View consistency check algorithm | RAL |
CHECK RESHARDING jobId | Execute resharding data consistency check | RAL |
START RESHARDING CHECK jobId | Start stopped resharding data consistency check job | RAL |
STOP RESHARDING CHECK jobId | Stopped resharding data consistency check job | RAL |
DROP RESHARDING CHECK jobId | Delete consistency check job | RAL |
COMMIT RESHARDING jobId | Commit resharding job | RAL |
FORCE COMMIT RESHARDING jobId | Force commit resharding job | RAL |
ROLLBACK RESHARDING jobId | Rollback resharding job | RAL |
APPLY RESHARDING jobId | Apply new resharding configurations (toggle metadata) | RAL |
FORCE APPLY RESHARDING jobId | Force apply new resharding configurations (toggle metadata) | RAL |
STOP RESHARDING SOURCE WRITING jobId | Stop writing | RAL |
RESTORE RESHARDING SOURCE WRITING jobId | Restore stop writing | RAL |
AUTO RESHARD TABLE tableName BY rule | Resharding auto mode | RDL |
RESHARD SINGLE TABLE tableName TO BROADCAST | Expand a single table into a broadcast table | RDL |
AUTO RESHARD SINGLE TABLE tableName TO BROADCAST | Single table expansion into broadcast table automatic mode | RDL |
1. Modify Resharding Rule #
copyALTER RESHARDING RULE
Example
copyALTER RESHARDING RULE (READ(RATE_LIMITER (TYPE(NAME='QPS’,PROPERTIES(‘qps'='5000’)))))
Config Description
copyALTER RESHARDING RULE (
READ( -- Data reading configuration. If it is not configured, part of the parameters will take effect by default.
WORKER_THREAD=20, -- Obtain the thread pool size of all the data from the source side. If it is not configured, the default value is used.
BATCH_SIZE=1000, -- The maximum number of records returned by a query operation. If it is not configured, the default value is used.
SHARDING_SIZE=10000000, -- Sharding size of all the data. If it is not configured, the default value is used.
RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
TYPE( -- Algorithm type. Option: QPS
NAME='QPS',
PROPERTIES( -- Algorithm property
'qps'='500'
)))
),
WRITE( -- Data writing configuration. If it is not configured, part of the parameters will take effect by default.
WORKER_THREAD=20, -- The size of the thread pool on which data is written into the target side. If it is not configured, the default value is used.
BATCH_SIZE=1000, -- The maximum number of records for a batch write operation. If it is not configured, the default value is used.
RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
TYPE( -- Algorithm type. Option: TPS
NAME='TPS',
PROPERTIES( -- Algorithm property.
'tps'='2000'
)))
),
STREAM_CHANNEL ( -- Data channel. It connects producers and consumers, used for reading and writing procedures. If it is not configured, the MEMORY type is used by default.
TYPE( -- Algorithm type. Option: MEMORY
NAME='MEMORY',
PROPERTIES( -- Algorithm property
'block-queue-size'='2000' -- Property: blocking queue size.
)))
);
2. Start Resharding #
copyRESHARD TABLE tableName BY rule
Parameter Description
Name | Description |
---|---|
tableName | Resharding table |
Example
copyRESHARD TABLE t_order BY(
STORAGE_UNITS(ds_0, ds_1, ds_2),
SHARDING_COLUMN=order_id,
TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);
3. View Resharding Rule #
copySHOW RESHARDING LIST;
Parameter Description
None
Example
copySHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id | tables | sharding_total_count | active | create_time | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2 | true | 2022-10-25 10:10:58 | NULL |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
4. View Resharding Job List #
copySHOW RESHARDING LIST;
Parameter Description
None
Example
copySHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id | tables | sharding_total_count | active | create_time | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2 | true | 2022-10-25 10:10:58 | NULL |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
5. View Resharding Job Status #
copySHOW RESHARDING STATUS jobId;
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copySHOW RESHARDING STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| item | data_source | status | active | processed_records_count | inventory_finished_percentage | remaining_seconds | incremental_idle_seconds | error_message |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| 0 | ds_0 | EXECUTE_INCREMENTAL_TASK | true | 3 | 100 | 0 | 92 | |
| 1 | ds_1 | EXECUTE_INCREMENTAL_TASK | true | 3 | 100 | 0 | 92 | |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
6. Start Stopped Resharding Job #
copySTART RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copySTART RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
7. Stop Resharding Job #
copySTOP RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copySTOP RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
8. View data consistency check algorithm #
For consistency check algorithm selection, please refer to Data Consistency Check Algorithm。
copySHOW RESHARDING CHECK ALGORITHMS
Example
copySHOW RESHARDING CHECK ALGORITHMS;
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| type | type_aliases | supported_database_types | description |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| CRC32_MATCH | | MySQL,MariaDB,H2,Aurora | Match CRC32 of records. |
| DATA_MATCH | | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Match raw data of records. |
| SphereEx:BATCH_CRC32_MATCH | SphereEx:DATA_CONSISTENCY_CHECK_CRC32_MATCH | MySQL,MariaDB,H2,Aurora | Match CRC32 of records (Commercial edition). |
| SphereEx:DATA_DIFF | | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Diff raw data of records. |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
Output Description
- type: algorithm name
- type_aliases: algorithm aliases
- supported_database_types: Which databases the algorithm supports
- description: Algorithm description
9. Check Resharding Data Consistency #
copyCHECK RESHARDING jobId BY TYPE (NAME='algorithmType');
Parameter Description
Name | Description |
---|---|
jobid | Job id |
algorithmType | Consistency check algorithm type |
Example
Specify the jobId to use specified verification algorithm to perform data consistency verification
copyCHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='CRC32_MATCH');
CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='DATA_MATCH', PROPERTIES('chunk-size'='1000'));
CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:BATCH_CRC32_MATCH');
CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:DATA_DIFF', PROPERTIES('diff-storage-type'='DATABASE','chunk-size'='1000','diff-storage-unit-name'='ds_2','diff-table-name'='spex_cnschk_1','incremental-idle-seconds-threshold'='30'));
10. Start Stopped Resharding Data Consistency Check Job #
copySTART RESHARDING CHECK jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copySTART RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';
11. Stop Resharding Data Consistency Check Job #
copySTOP RESHARDING CHECK jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copySTOP RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';
12. Delete Resharding Data Consistency Check Job #
copyDROP RESHARDING CHECK jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copyDROP RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';
13. Commit Resharding Job #
copyCOMMIT RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copyCOMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
14. Force Commit Resharding Job #
copyFORCE COMMIT RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copyFORCE COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
15. Rollback Resharding Job #
copyROLLBACK RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copyROLLBACK RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
16. Apply New Resharding Configurations (toggle metadata) #
copyAPPLY RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copyAPPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
17. Force Apply New Resharding Configurations (toggle metadata) #
copyFORCE APPLY RESHARDING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copyFORCE APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';
18. Stop Writing #
copySTOP RESHARDING SOURCE WRITING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copySTOP RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';
19. Restore Stop Writing #
copyRESTORE RESHARDING SOURCE WRITING jobId
Parameter Description
Name | Description |
---|---|
jobid | Job id |
Example
copyRESTORE RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';
20. Resharding Auto Mode #
copyAUTO RESHARD TABLE tableName BY rule
Parameter Description
Name | Description |
---|---|
tableName | Resharding table |
Example
copyAUTO RESHARD TABLE t_order BY(
STORAGE_UNITS(ds_0, ds_1, ds_2),
SHARDING_COLUMN=order_id,
TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);
21. Expand a single table into a broadcast table #
copyRESHARD SINGLE TABLE tableName TO BROADCAST
Parameter Description
Name | Description |
---|---|
tableName | Resharding single table |
Example
copyRESHARD SINGLE TABLE t_address TO BROADCAST
22. Single table expansion into broadcast table automatic mode #
copyAUTO RESHARD SINGLE TABLE tableName TO BROADCAST
Parameter Description
Name | Description |
---|---|
tableName | Resharding single table |
Example
copyAUTO RESHARD SINGLE TABLE t_address TO BROADCAST