Data consistency check algorithm #
CRC32 consistency check #
Type: CRC32_MATCH
Configurable properties
None
Note: Applicable to MySQL and derivative databases, only existing data, writing needs to be stopped, and matching CRC32 of all records.
Usage example
CHECK MIGRATION jobId BY TYPE (NAME='CRC32_MATCH');
CHECK RESHARDING jobId BY TYPE (NAME='CRC32_MATCH');
Data matching consistency check #
Type: DATA_MATCH
Configurable properties:
| Properties name | Data type | Default value | Description |
|---|---|---|---|
| chunk-size | int | 1000 | The maximum number of records returned by a query operation |
Note: Applicable to all databases, only existing data, writing needs to be stopped, matching all field values of all records.
Usage example :
CHECK MIGRATION jobId BY TYPE (NAME='DATA_MATCH', PROPERTIES('chunk-size'='1000'));
CHECK RESHARDING jobId BY TYPE (NAME='DATA_MATCH', PROPERTIES('chunk-size'='1000'));
CRC32 Consistency check (commercial version) #
Type: SphereEx:BATCH_CRC32_MATCH, SphereEx:DATA_CONSISTENCY_CHECK_CRC32_MATCH
Configurable properties:
None
Note: Applicable to MySQL and derivative databases, only existing data. It is necessary to stop writing and batch match the CRC32 of all records.
Usage example:
CHECK MIGRATION jobId BY TYPE (NAME='SphereEx:BATCH_CRC32_MATCH');
CHECK RESHARDING jobId BY TYPE (NAME='SphereEx:BATCH_CRC32_MATCH');
Real-time consistency verification (commercial version) #
experimental feature
Type: SphereEx:DATA_DIFF
Configurable properties:
| Properties name | Data type | Default value | Description |
|---|---|---|---|
| chunk-size | int | 1000 | The maximum number of records returned by a query operation |
| diff-storage-type | String | Data difference storage media type. Value: DATABASE (stored to a proxy storage unit) | |
| diff-storage-unit-name | String | Which proxy storage unit is the data difference stored in, such as ds_0. Required when diff-storage-type is DATABASE | |
| diff-table-name | String | In which table the data differences are stored (make sure the table name is not occupied). Required when diff-storage-type is DATABASE | |
| incremental-idle-seconds-threshold | int | 5 | The time threshold (unit: second) when incremental data is no longer active. If this time is exceeded, it is considered that there is no incremental data in this round. Required when diff-storage-type is DATABASE |
Description: Applicable to all databases, supports existing data and incremental data, no need to stop writing, dynamically matches all field values of all records, the data difference between the source and target ends will be recorded to the configured storage medium (usually a difference table) When no incremental data and no differential data are detected in the final 3 rounds, the verification is passed.
Usage example:
CHECK MIGRATION jobId BY TYPE (NAME='SphereEx:DATA_DIFF', PROPERTIES('diff-storage-type'='DATABASE','chunk-size'='1000','diff-storage-unit-name'='ds_2','diff-table-name'='spex_cnschk_1','incremental-idle-seconds-threshold'='30'));
CHECK RESHARDING jobId BY TYPE (NAME='SphereEx:DATA_DIFF', PROPERTIES('diff-storage-type'='DATABASE','chunk-size'='1000','diff-storage-unit-name'='ds_2','diff-table-name'='spex_cnschk_1','incremental-idle-seconds-threshold'='30'));
The above example represents real-time consistency verification for data migration and expansion and contraction respectively, and is stored in the proxy ds_2 storage unit and spex_cnschk_1 table. The time threshold for incremental data to be inactive is 30 seconds.
Advantage:
- Works with all databases
- No need to stop writing