Logo
Resharding

Resharding #

SyntaxDescriptionType
ALTER RESHARDING RULEModify resharding ruleRAL
RESHARD TABLE tableName by ruleStart reshardingRDL
SHOW RESHARDING RULEView resharding ruleRAL
SHOW RESHARDING LISTView resharding job listRAL
SHOW RESHARDING STATUS jobIdView resharding job statusRAL
START RESHARDING jobIdStart stopped resharding jobRAL
STOP RESHARDING jobIdStop resharding jobRAL
SHOW RESHARDING CHECK ALGORITHMSView consistency check algorithmRAL
CHECK RESHARDING jobIdExecute resharding data consistency checkRAL
START RESHARDING CHECK jobIdStart stopped resharding data consistency check jobRAL
STOP RESHARDING CHECK jobIdStopped resharding data consistency check jobRAL
DROP RESHARDING CHECK jobIdDelete consistency check jobRAL
COMMIT RESHARDING jobIdCommit resharding jobRAL
FORCE COMMIT RESHARDING jobIdForce commit resharding jobRAL
ROLLBACK RESHARDING jobIdRollback resharding jobRAL
APPLY RESHARDING jobIdApply new resharding configurations (toggle metadata)RAL
FORCE APPLY RESHARDING jobIdForce apply new resharding configurations (toggle metadata)RAL
STOP RESHARDING SOURCE WRITING jobIdStop writingRAL
RESTORE RESHARDING SOURCE WRITING jobIdRestore stop writingRAL
AUTO RESHARD TABLE tableName BY ruleResharding auto modeRDL
RESHARD SINGLE TABLE tableName TO BROADCASTExpand a single table into a broadcast tableRDL
AUTO RESHARD SINGLE TABLE tableName TO BROADCASTSingle table expansion into broadcast table automatic modeRDL

1. Modify Resharding Rule #

ALTER RESHARDING RULE 

Example

ALTER RESHARDING RULE (READ(RATE_LIMITER (TYPE(NAME='QPS’,PROPERTIES(‘qps'='5000’)))))

Config Description

ALTER RESHARDING RULE (
READ( -- Data reading configuration. If it is not configured, part of the parameters will take effect by default.
  WORKER_THREAD=20, -- Obtain the thread pool size of all the data from the source side. If it is not configured, the default value is used.
  BATCH_SIZE=1000, -- The maximum number of records returned by a query operation. If it is not configured, the default value is used.
  SHARDING_SIZE=10000000, -- Sharding size of all the data. If it is not configured, the default value is used.
  RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
  TYPE( -- Algorithm type. Option: QPS
  NAME='QPS',
  PROPERTIES( -- Algorithm property
  'qps'='500'
  )))
),
WRITE( -- Data writing configuration. If it is not configured, part of the parameters will take effect by default.
  WORKER_THREAD=20, -- The size of the thread pool on which data is written into the target side. If it is not configured, the default value is used.
  BATCH_SIZE=1000, -- The maximum number of records for a batch write operation. If it is not configured, the default value is used.
  RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
  TYPE( -- Algorithm type. Option: TPS
  NAME='TPS',
  PROPERTIES( -- Algorithm property.
  'tps'='2000'
  )))
),
STREAM_CHANNEL ( -- Data channel. It connects producers and consumers, used for reading and writing procedures. If it is not configured, the MEMORY type is used by default.
TYPE( -- Algorithm type. Option: MEMORY
NAME='MEMORY',
PROPERTIES( -- Algorithm property
'block-queue-size'='2000' -- Property: blocking queue size.
)))
);

2. Start Resharding #

RESHARD TABLE tableName BY rule 

Parameter Description

NameDescription
tableNameResharding table

Example

RESHARD TABLE t_order BY(
    STORAGE_UNITS(ds_0, ds_1, ds_2),
    SHARDING_COLUMN=order_id,
    TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
    KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

3. View Resharding Rule #

SHOW RESHARDING LIST;

Parameter Description

None

Example

SHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id                                    | tables  | sharding_total_count | active | create_time         | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2                    | true   | 2022-10-25 10:10:58 | NULL      |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+

4. View Resharding Job List #

SHOW RESHARDING LIST;

Parameter Description

None

Example

SHOW RESHARDING LIST;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id                                    | tables  | sharding_total_count | active | create_time         | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2                    | true   | 2022-10-25 10:10:58 | NULL      |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+

5. View Resharding Job Status #

SHOW RESHARDING STATUS jobId;

Parameter Description

NameDescription
jobidJob id

Example

SHOW RESHARDING STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| item | data_source | status                   | active | processed_records_count | inventory_finished_percentage | remaining_seconds | incremental_idle_seconds | error_message |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| 0    | ds_0        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
| 1    | ds_1        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+

6. Start Stopped Resharding Job #

START RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

START RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

7. Stop Resharding Job #

STOP RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

STOP RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

8. View data consistency check algorithm #

For consistency check algorithm selection, please refer to Data Consistency Check Algorithm

SHOW RESHARDING CHECK ALGORITHMS 

Example

SHOW RESHARDING CHECK ALGORITHMS;
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| type                       | type_aliases                                | supported_database_types                                                                      | description                                  |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+
| CRC32_MATCH                |                                             | MySQL,MariaDB,H2,Aurora                                                                       | Match CRC32 of records.                      |
| DATA_MATCH                 |                                             | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Match raw data of records.                   |
| SphereEx:BATCH_CRC32_MATCH | SphereEx:DATA_CONSISTENCY_CHECK_CRC32_MATCH | MySQL,MariaDB,H2,Aurora                                                                       | Match CRC32 of records (Commercial edition). |
| SphereEx:DATA_DIFF         |                                             | SQL92,MySQL,PostgreSQL,openGauss,Oracle,SQLServer,MariaDB,H2,Hive,Presto,Aurora,DM,KingbaseES | Diff raw data of records.                    |
+----------------------------+---------------------------------------------+-----------------------------------------------------------------------------------------------+----------------------------------------------+

Output Description

  • type: algorithm name
  • type_aliases: algorithm aliases
  • supported_database_types: Which databases the algorithm supports
  • description: Algorithm description

9. Check Resharding Data Consistency #

CHECK RESHARDING jobId BY TYPE (NAME='algorithmType');

Parameter Description

NameDescription
jobidJob id
algorithmTypeConsistency check algorithm type

Example

Specify the jobId to use specified verification algorithm to perform data consistency verification

CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='CRC32_MATCH');

CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='DATA_MATCH', PROPERTIES('chunk-size'='1000'));

CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:BATCH_CRC32_MATCH');

CHECK RESHARDING 'j01016e501b498ed1bdb2c373a2e85e2529a6' BY TYPE (NAME='SphereEx:DATA_DIFF', PROPERTIES('diff-storage-type'='DATABASE','chunk-size'='1000','diff-storage-unit-name'='ds_2','diff-table-name'='spex_cnschk_1','incremental-idle-seconds-threshold'='30'));

10. Start Stopped Resharding Data Consistency Check Job #

START RESHARDING CHECK jobId

Parameter Description

NameDescription
jobidJob id

Example

START RESHARDING CHECK  'j51017f973ac82cb1edea4f5238a258c25e89';

11. Stop Resharding Data Consistency Check Job #

STOP RESHARDING CHECK jobId

Parameter Description

NameDescription
jobidJob id

Example

STOP RESHARDING CHECK  'j51017f973ac82cb1edea4f5238a258c25e89';

12. Delete Resharding Data Consistency Check Job #

DROP RESHARDING CHECK jobId

Parameter Description

NameDescription
jobidJob id

Example

DROP RESHARDING CHECK 'j51017f973ac82cb1edea4f5238a258c25e89';

13. Commit Resharding Job #

COMMIT RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

14. Force Commit Resharding Job #

FORCE COMMIT RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

FORCE COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

15. Rollback Resharding Job #

ROLLBACK RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

ROLLBACK RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

16. Apply New Resharding Configurations (toggle metadata) #

APPLY RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

17. Force Apply New Resharding Configurations (toggle metadata) #

FORCE APPLY RESHARDING jobId

Parameter Description

NameDescription
jobidJob id

Example

FORCE APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

18. Stop Writing #

STOP RESHARDING SOURCE WRITING jobId

Parameter Description

NameDescription
jobidJob id

Example

STOP RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

19. Restore Stop Writing #

RESTORE RESHARDING SOURCE WRITING jobId

Parameter Description

NameDescription
jobidJob id

Example

RESTORE RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

20. Resharding Auto Mode #

AUTO RESHARD TABLE tableName BY rule

Parameter Description

NameDescription
tableNameResharding table

Example

AUTO RESHARD TABLE t_order BY(
    STORAGE_UNITS(ds_0, ds_1, ds_2),
    SHARDING_COLUMN=order_id,
    TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
    KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

21. Expand a single table into a broadcast table #

RESHARD SINGLE TABLE tableName TO BROADCAST

Parameter Description

NameDescription
tableNameResharding single table

Example

RESHARD SINGLE TABLE t_address TO BROADCAST

22. Single table expansion into broadcast table automatic mode #

AUTO RESHARD SINGLE TABLE tableName TO BROADCAST

Parameter Description

NameDescription
tableNameResharding single table

Example

AUTO RESHARD SINGLE TABLE t_address TO BROADCAST