Logo
Data Migration

Data Migration #

Background #

In a scenario where the business continues to develop and the amount of data and concurrency reaches a certain extent, the traditional single database may face problems in terms of performance, scalability and availability.

Although NoSQL solutions can solve the above problems through data sharding and horizontal scale-out, NoSQL databases generally do not support transactions and SQL.

DBPlusEngine can also solve the above problems and supports data sharding and horizontal scale-out, while at the same time, also supporting distributed transactions and SQL.

The data migration scheme provided by DBPlusEngine can help the traditional single database smoothly switch to DBPlusEngine.

Challenges #

The data migration process should not affect the running services. So the first challenge is to minimize the time window during which data is not available.

Next, data migration should not affect existing data. So the second challenge is to ensure the data correctness.

Goal #

The major goal of SphereEx-DBPlusEngine in performing data migration is to reduce the impact of data migration on services and provide a one-stop universal data migration solution.

Core Concept #

Nodes #

Instances for running compute or storage tier component processes. These can either be physical machines, virtual machines, or containers, etc.

Cluster #

Multiple nodes that are assembled together to provide a specified service.

Source #

The storage cluster where the original data resides.

Target #

The target storage cluster to which the original data is to be migrated.

Data Migration Process #

The entire process of replicating data from one storage cluster to another.

Stock Data #

The data that was already in the data node before the data migration operation started.

Incremental Data #

New data generated by operational systems during the execution of data migration operations.

Limitations #

Supported Procedures #

  • Migration of peripheral data to databases managed by SphereEx-DBPlusEngine.
  • Migration of integer or string primary key tables.

Procedures not supported #

  • Migrate tables without primary keys or unique keys.
  • Migrate tables with composite primary keys or composite unique keys.
  • Migration on top of the current storage node is not supported, so a brand new database cluster needs to be prepared as the migration target cluster.