Logo
Resharding

Resharding #

SyntaxDescriptionType
ALTER RESHARDING RULEModify resharding ruleRAL
RESHARD TABLE tableName by ruleStart reshardingRDL
SHOW RESHARDING RULEView resharding ruleRAL
SHOW RESHARDING LISTView resharding job listRAL
SHOW RESHARDING STATUS jobIdView resharding job statusRAL
START RESHARDING jobIdStart stopped resharding jobRAL
STOP RESHARDING jobIdStop resharding jobRAL
SHOW RESHARDING CHECK ALGORITHMSView consistency check algorithmRAL
CHECK RESHARDING jobIdExecute resharding data consistency checkRAL
START RESHARDING CHECK jobIdStart stopped resharding data consistency check jobRAL
STOP RESHARDING CHECK jobIdStopped resharding data consistency check jobRAL
DROP RESHARDING CHECK jobIdDelete consistency check jobRAL
COMMIT RESHARDING jobIdCommit resharding jobRAL
FORCE COMMIT RESHARDING jobIdForce commit resharding jobRAL
ROLLBACK RESHARDING jobIdRollback resharding jobRAL
APPLY RESHARDING jobIdApply new resharding configurations (toggle metadata)RAL
FORCE APPLY RESHARDING jobIdForce apply new resharding configurations (toggle metadata)RAL
STOP RESHARDING SOURCE WRITING jobIdStop writingRAL
RESTORE RESHARDING SOURCE WRITING jobIdRestore stop writingRAL
AUTO RESHARD TABLE tableName BY ruleResharding auto modeRDL
RESHARD SINGLE TABLE tableName TO BROADCASTExpand a single table into a broadcast tableRDL
AUTO RESHARD SINGLE TABLE tableName TO BROADCASTSingle table expansion into broadcast table automatic modeRDL

1. Modify Resharding Rule #

copyALTER RESHARDING RULE 

Example

copyALTER RESHARDING RULE (READ(RATE_LIMITER (TYPE(NAME='QPS’,PROPERTIES(‘qps'='5000’)))))

Config Description

copyALTER RESHARDING RULE (
READ( -- Data reading configuration. If it is not configured, part of the parameters will take effect by default.
  WORKER_THREAD=20, -- Obtain the thread pool size of all the data from the source side. If it is not configured, the default value is used.
  BATCH_SIZE=1000, -- The maximum number of records returned by a query operation. If it is not configured, the default value is used.
  SHARDING_SIZE=10000000, -- Sharding size of all the data. If it is not configured, the default value is used.
  RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
  TYPE( -- Algorithm type. Option: QPS
  NAME='QPS',
  PROPERTIES( -- Algorithm property
  'qps'='500'
  )))
),
WRITE( -- Data writing configuration. If it is not configured, part of the parameters will take effect by default.
  WORKER_THREAD=20, -- The size of the thread pool on which data is written into the target side. If it is not configured, the default value is used.
  BATCH_SIZE=1000, -- The maximum number of records for a batch write operation. If it is not configured, the default value is used.
  RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
  TYPE( -- Algorithm type. Option: TPS
  NAME='TPS',
  PROPERTIES( -- Algorithm property.
  'tps'='2000'
  )))
),
STREAM_CHANNEL ( -- Data channel. It connects producers and consumers, used for reading and writing procedures. If it is not configured, the MEMORY type is used by default.
TYPE( -- Algorithm type. Option: MEMORY
NAME='MEMORY',
PROPERTIES( -- Algorithm property
'block-queue-size'='2000' -- Property: blocking queue size.
)))
);

2. Start Resharding #

copyRESHARD TABLE tableName BY rule 

Parameter Description

NameDescription
tableNameResharding table

Example

copyRESHARD TABLE t_order BY(
    STORAGE_UNITS(ds_0, ds_1, ds_2),
    SHARDING_COLUMN=order_id,
    TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
    KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

3. View Resharding Rule #

copySHOW RESHARDING LIST;

Parameter Description

None

Example

copySHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id                                    | tables  | sharding_total_count | active | create_time         | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2                    | true   | 2022-10-25 10:10:58 | NULL      |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+

4. View Resharding Job List #

copySHOW RESHARDING LIST;

Parameter Description

None

Example

copySHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id                                    | tables  | sharding_total_count | active | create_time         | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2                    | true   | 2022-10-25 10:10:58 | NULL      |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+

5. View Resharding Job Status #

copySHOW RESHARDING STATUS jobId;

Parameter Description

NameDescription
jobidJob id

Example

copySHOW RESHARDING STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| item | data_source | status                   | active | processed_records_count | inventory_finished_percentage | remaining_seconds | incremental_idle_seconds | error_message |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| 0    | ds_0        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
| 1    | ds_1        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+

6. Start Stopped Resharding Job #

copySTART RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

copySTART RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

7. Stop Resharding Job #

copySTOP RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

copySTOP RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

8. View data consistency check algorithm #

For consistency check algorithm selection, please refer to Data Consistency Check Algorithm

copySHOW RESHARDING CHECK ALGORITHMS 

Example

copySHOW RESHARDING CHECK ALGORITHMS;
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| type                       | type_aliases                                | supported_database_types                                                                      | description                                  |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| CRC32_MATCH                |                                             | MySQL,MariaDB,H2,Aurora                                                                       | Match CRC32 of records.                      |
| DATA_MATCH                 |                                             | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Match raw data of records.                   |
| SphereEx:BATCH_CRC32_MATCH | SphereEx:DATA_CONSISTENCY_CHECK_CRC32_MATCH | MySQL,MariaDB,H2,Aurora                                                                       | Match CRC32 of records (Commercial edition). |
| SphereEx:DATA_DIFF         |                                             | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Diff raw data of records.                    |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+

Output Description

  • type: algorithm name
  • type_aliases: algorithm aliases
  • supported_database_types: Which databases the algorithm supports
  • description: Algorithm description

9. Check Resharding Data Consistency #

copyCHECK RESHARDING jobId BY TYPE (NAME='algorithmType');

Parameter Description

NameDescription
jobidJob id
algorithmTypeConsistency check algorithm type

Example

Specify the jobId to use specified verification algorithm to perform data consistency verification

copyCHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='CRC32_MATCH');

CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='DATA_MATCH', PROPERTIES('chunk-size'='1000'));

CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:BATCH_CRC32_MATCH');

CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:DATA_DIFF', PROPERTIES('diff-storage-type'='DATABASE','chunk-size'='1000','diff-storage-unit-name'='ds_2','diff-table-name'='spex_cnschk_1','incremental-idle-seconds-threshold'='30'));

10. Start Stopped Resharding Data Consistency Check Job #

copySTART RESHARDING CHECK jobId

Parameter Description

NameDescription
jobidJob id

Example

copySTART RESHARDING CHECK  'j51017f973ac82cb1edea4f5238a258c25e89';

11. Stop Resharding Data Consistency Check Job #

copySTOP RESHARDING CHECK jobId

Parameter Description

NameDescription
jobidJob id

Example

copySTOP RESHARDING CHECK  'j51017f973ac82cb1edea4f5238a258c25e89';

12. Delete Resharding Data Consistency Check Job #

copyDROP RESHARDING CHECK jobId

Parameter Description

NameDescription
jobidJob id

Example

copyDROP RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';

13. Commit Resharding Job #

copyCOMMIT RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

copyCOMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

14. Force Commit Resharding Job #

copyFORCE COMMIT RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

copyFORCE COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

15. Rollback Resharding Job #

copyROLLBACK RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

copyROLLBACK RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

16. Apply New Resharding Configurations (toggle metadata) #

copyAPPLY RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

copyAPPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

17. Force Apply New Resharding Configurations (toggle metadata) #

copyFORCE APPLY RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

copyFORCE APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

18. Stop Writing #

copySTOP RESHARDING SOURCE WRITING jobId

Parameter Description

NameDescription
jobidJob id

Example

copySTOP RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

19. Restore Stop Writing #

copyRESTORE RESHARDING SOURCE WRITING jobId

Parameter Description

NameDescription
jobidJob id

Example

copyRESTORE RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

20. Resharding Auto Mode #

copyAUTO RESHARD TABLE tableName BY rule

Parameter Description

NameDescription
tableNameResharding table

Example

copyAUTO RESHARD TABLE t_order BY(
    STORAGE_UNITS(ds_0, ds_1, ds_2),
    SHARDING_COLUMN=order_id,
    TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
    KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

21. Expand a single table into a broadcast table #

copyRESHARD SINGLE TABLE tableName TO BROADCAST

Parameter Description

NameDescription
tableNameResharding single table

Example

copyRESHARD SINGLE TABLE t_address TO BROADCAST

22. Single table expansion into broadcast table automatic mode #

copyAUTO RESHARD SINGLE TABLE tableName TO BROADCAST

Parameter Description

NameDescription
tableNameResharding single table

Example

copyAUTO RESHARD SINGLE TABLE t_address TO BROADCAST