使用手册 #

MySQL 使用手册 #

环境要求 #

支持的 MySQL 版本：5.1.15 ~ 8.0.x。

权限要求 #

开启 binlog。

MySQL 5.7 my.cnf 示例配置：

[mysqld]
server-id=1
log-bin=mysql-bin
binlog-format=row
binlog-row-image=full
max_connections=600

执行以下命令，确认是否有开启 binlog：

show variables like '%log_bin%';
show variables like '%binlog%';

如以下显示，则说明 binlog 已开启

+-----------------------------------------+---------------------------------------+
| Variable_name                           | Value                                 |
+-----------------------------------------+---------------------------------------+
| log_bin                                 | ON                                    |
| binlog_format                           | ROW                                   |
| binlog_row_image                        | FULL                                  |
+-----------------------------------------+---------------------------------------+

赋予 MySQL 账号 Replication 相关权限。

执行以下命令，查看该用户是否有迁移权限：

SHOW GRANTS FOR 'user';

示例结果：

+------------------------------------------------------------------------------+
|Grants for ${username}@${host}                                                |
+------------------------------------------------------------------------------+
|GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO ${username}@${host}     |
|.......                                                                       |
+------------------------------------------------------------------------------+

完整流程示例 #

前提条件 #

在 MySQL 中创建好数据库。

示例：

DROP DATABASE IF EXISTS resharding_ds_0;
CREATE DATABASE resharding_ds_0 DEFAULT CHARSET utf8;
    
DROP DATABASE IF EXISTS resharding_ds_1;
CREATE DATABASE resharding_ds_1 DEFAULT CHARSET utf8;
    
DROP DATABASE IF EXISTS resharding_ds_2;
CREATE DATABASE resharding_ds_2 DEFAULT CHARSET utf8;

操作步骤 #

在 proxy 新建逻辑数据库并配置好资源和规则，建表并初始化一些数据。

CREATE DATABASE sharding_db;

USE sharding_db;

REGISTER STORAGE UNIT ds_0 (
    URL="jdbc:mysql://127.0.0.1:3306/resharding_ds_0?useServerPrepStmts=true&serverTimezone=UTC&useSSL=false&characterEncoding=utf-8",
    USER="root",
    PASSWORD="root"
), ds_1 (
    URL="jdbc:mysql://127.0.0.1:3306/resharding_ds_1?useServerPrepStmts=true&serverTimezone=UTC&useSSL=false&characterEncoding=utf-8",
    USER="root",
    PASSWORD="root"
);
    
CREATE SHARDING TABLE RULE t_order(
STORAGE_UNITS(ds_0,ds_1),
SHARDING_COLUMN=order_id,
TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="4")),
KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

CREATE TABLE t_order (order_id INT NOT NULL, user_id INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (order_id));

INSERT INTO t_order (order_id, user_id, status) VALUES (1,2,'ok'),(2,4,'ok'),(3,6,'ok'),(4,1,'ok'),(5,3,'ok'),(6,5,'ok');

在 Proxy 中添加新数据源。

示例：

REGISTER STORAGE UNIT ds_2 (
    URL="jdbc:mysql://127.0.0.1:3306/resharding_ds_2?useServerPrepStmts=true&serverTimezone=UTC&useSSL=false&characterEncoding=utf-8",
    USER="root",
    PASSWORD="root"
);

启动扩缩容。

新增了 ds_2 数据源，示例：

RESHARD TABLE t_order BY(
    STORAGE_UNITS(ds_0, ds_1, ds_2),
    SHARDING_COLUMN=order_id,
    TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
    KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

查看扩缩容作业列表。

SHOW RESHARDING LIST;

示例结果：

+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id                                    | tables  | sharding_total_count | active | create_time         | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2                    | true   | 2022-10-25 10:10:58 | NULL      |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+

查看扩缩容详情。

SHOW RESHARDING STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';

示例结果：

+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| item | data_source | status                   | active | processed_records_count | inventory_finished_percentage | remaining_seconds | incremental_idle_seconds | error_message |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| 0    | ds_0        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
| 1    | ds_1        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+

执行停写（可选）。

该语句会拦截增删改和部分DistSQL，是可选的。

STOP RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

执行数据一致性校验。

CHECK RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89' BY TYPE (NAME='CRC32_MATCH');

数据一致性校验算法类型来自：

SHOW RESHARDING CHECK ALGORITHMS;
+-------------+--------------------------------------------------------------+----------------------------+
| type        | supported_database_types                                     | description                |
+-------------+--------------------------------------------------------------+----------------------------+
| CRC32_MATCH | MySQL                                                        | Match CRC32 of records.    |
| DATA_MATCH  | SQL92,MySQL,MariaDB,PostgreSQL,openGauss,Oracle,SQLServer,H2 | Match raw data of records. |
+-------------+--------------------------------------------------------------+----------------------------+

开启数据加密的情况需要使用 DATA_MATCH。

查询数据一致性校验进度：

SHOW RESHARDING CHECK STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';

示例结果：

+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+
| tables  | result | finished_percentage | remaining_seconds | check_begin_time        | check_end_time          | duration_seconds | error_message |
+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+
| t_order | true   | 100                 | 0                 | 2022-10-25 10:13:33.220 | 2022-10-25 10:13:35.338 | 2                |               |
+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+

切换元数据。

APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

预览生效后的规则

PREVIEW SELECT * FROM t_order;

示例结果：

+------------------+-----------------------------------------------------------------+
| data_source_name | actual_sql                                                      |
+------------------+-----------------------------------------------------------------+
| ds_0             | select * from v1_t_order_0 UNION ALL select * from v1_t_order_3 |
| ds_1             | select * from v1_t_order_1 UNION ALL select * from v1_t_order_4 |
| ds_2             | select * from v1_t_order_2 UNION ALL select * from v1_t_order_5 |
+------------------+-----------------------------------------------------------------+

恢复停写（可选）。

如果在之前执行了停写，需要进行恢复。

RESTORE RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

完成扩缩容作业。

示例：

COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

PostgreSQL 使用手册 #

环境要求 #

支持的 PostgreSQL 版本：9.4 或以上版本。

权限要求 #

开启 test_decoding。
调整 WAL 配置。

postgresql.conf 示例配置：

wal_level = logical
max_wal_senders = 10
max_replication_slots = 10
max_connections = 600

详情请参见 Write Ahead Log 和 Replication。

配置 PostgreSQL 允许 Proxy 拥有 replication 权限。

pg_hba.conf 示例配置：

host replication repl_acct 0.0.0.0/0 md5

详情请参见 The pg_hba.conf File。

完整流程示例 #

前提条件 #

在 PostgreSQL 中创建好数据库。

示例：

DROP DATABASE IF EXISTS resharding_ds_0;
CREATE DATABASE resharding_ds_0;

DROP DATABASE IF EXISTS resharding_ds_1;
CREATE DATABASE resharding_ds_1;
    
DROP DATABASE IF EXISTS resharding_ds_2;
CREATE DATABASE resharding_ds_2;

操作步骤 #

在 proxy 新建逻辑数据库并配置好资源和规则，建表并初始化一些数据。

CREATE DATABASE sharding_db;

\c sharding_db;

REGISTER STORAGE UNIT ds_0 (
    URL="jdbc:postgresql://127.0.0.1:5432/resharding_ds_0",
    USER="root",
    PASSWORD="root"
), ds_1 (
    URL="jdbc:postgresql://127.0.0.1:5432/resharding_ds_1",
    USER="root",
    PASSWORD="root"
);
    
CREATE SHARDING TABLE RULE t_order(
STORAGE_UNITS(ds_0,ds_1),
SHARDING_COLUMN=order_id,
TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="4")),
KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

CREATE TABLE t_order (order_id INT NOT NULL, user_id INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (order_id));

INSERT INTO t_order (order_id, user_id, status) VALUES (1,2,'ok'),(2,4,'ok'),(3,6,'ok'),(4,1,'ok'),(5,3,'ok'),(6,5,'ok');

在 Proxy 中添加新数据源。

示例：

REGISTER STORAGE UNIT ds_2 (
    URL="jdbc:postgresql://127.0.0.1:5432/resharding_ds_2",
    USER="root",
    PASSWORD="root"
);

启动扩缩容。

新增了 ds_2 数据源，示例：

RESHARD TABLE t_order BY(
    STORAGE_UNITS(ds_0, ds_1, ds_2),
    SHARDING_COLUMN=order_id,
    TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
    KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

查看扩缩容作业列表。

SHOW RESHARDING LIST;

示例结果：

+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id                                    | tables  | sharding_total_count | active | create_time         | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2                    | true   | 2022-10-25 10:10:58 | NULL      |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+

查看扩缩容详情。

SHOW RESHARDING STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';

示例结果：

+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| item | data_source | status                   | active | processed_records_count | inventory_finished_percentage | remaining_seconds | incremental_idle_seconds | error_message |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| 0    | ds_0        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
| 1    | ds_1        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+

执行停写（可选）。

该语句会拦截增删改和部分DistSQL，是可选的。

STOP RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

执行数据一致性校验。

CHECK RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

查询数据一致性校验进度：

SHOW RESHARDING CHECK STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';

示例结果：

+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+
| tables  | result | finished_percentage | remaining_seconds | check_begin_time        | check_end_time          | duration_seconds | error_message |
+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+
| t_order | true   | 100                 | 0                 | 2022-10-25 10:13:33.220 | 2022-10-25 10:13:35.338 | 2                |               |
+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+

切换元数据。

APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

预览生效后的规则：

PREVIEW SELECT * FROM t_order;

示例结果：

+------------------+-----------------------------------------------------------------+
| data_source_name | actual_sql                                                      |
+------------------+-----------------------------------------------------------------+
| ds_0             | select * from v1_t_order_0 UNION ALL select * from v1_t_order_3 |
| ds_1             | select * from v1_t_order_1 UNION ALL select * from v1_t_order_4 |
| ds_2             | select * from v1_t_order_2 UNION ALL select * from v1_t_order_5 |
+------------------+-----------------------------------------------------------------+

恢复停写（可选）

如果在之前执行了停写，需要进行恢复。

RESTORE RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

完成扩缩容作业。

示例：

COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

openGauss 使用手册 #

环境要求 #

支持的 openGauss 版本：2.0.1 ~ 3.0.0。

权限要求 #

调整 WAL 配置。

postgresql.conf 示例配置：

wal_level = logical
max_wal_senders = 10
max_replication_slots = 10
wal_sender_timeout = 0
max_connections = 600

详情请参见 Write Ahead Log 和 Replication。

配置 PostgreSQL 允许 Proxy 拥有 replication 权限。

pg_hba.conf 示例配置：

host replication repl_acct 0.0.0.0/0 md5

详情请参见 Configuring Client Access Authentication 和 Example: Logic Replication Code。

完整流程示例 #

前提条件 #

在 openGauss 中创建好数据库。

示例：

DROP DATABASE IF EXISTS resharding_ds_0;
CREATE DATABASE resharding_ds_0;

DROP DATABASE IF EXISTS resharding_ds_1;
CREATE DATABASE resharding_ds_1;
    
DROP DATABASE IF EXISTS resharding_ds_2;
CREATE DATABASE resharding_ds_2;

操作步骤 #

在 proxy 新建逻辑数据库并配置好资源和规则，建表并初始化一些数据。

CREATE DATABASE sharding_db;

USE sharding_db;

REGISTER STORAGE UNIT ds_0 (
    URL="jdbc:opengauss://127.0.0.1:5432/resharding_ds_0",
    USER="root",
    PASSWORD="root"
), ds_1 (
    URL="jdbc:opengauss://127.0.0.1:5432/resharding_ds_1",
    USER="root",
    PASSWORD="root"
);
    
CREATE SHARDING TABLE RULE t_order(
STORAGE_UNITS(ds_0,ds_1),
SHARDING_COLUMN=order_id,
TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="4")),
KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

CREATE TABLE t_order (order_id INT NOT NULL, user_id INT NOT NULL, status VARCHAR(45) NULL, PRIMARY KEY (order_id));

INSERT INTO t_order (order_id, user_id, status) VALUES (1,2,'ok'),(2,4,'ok'),(3,6,'ok'),(4,1,'ok'),(5,3,'ok'),(6,5,'ok');

在 Proxy 中添加新数据源。

示例：

REGISTER STORAGE UNIT ds_2 (
    URL="jdbc:opengauss://127.0.0.1:5432/resharding_ds_2",
    USER="root",
    PASSWORD="root"
);

启动扩缩容。

新增了 ds_2 数据源，示例：

RESHARD TABLE t_order BY(
    STORAGE_UNITS(ds_0, ds_1, ds_2),
    SHARDING_COLUMN=order_id,
    TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="6")),
    KEY_GENERATE_STRATEGY(COLUMN=order_id,TYPE(NAME="snowflake"))
);

查看扩缩容作业列表。

SHOW RESHARDING LIST;

示例结果：

+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id                                    | tables  | sharding_total_count | active | create_time         | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j51017f973ac82cb1edea4f5238a258c25e89 | t_order | 2                    | true   | 2022-10-25 10:10:58 | NULL      |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+

查看扩缩容详情。

SHOW RESHARDING STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';

示例结果：

+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| item | data_source | status                   | active | processed_records_count | inventory_finished_percentage | remaining_seconds | incremental_idle_seconds | error_message |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+
| 0    | ds_0        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
| 1    | ds_1        | EXECUTE_INCREMENTAL_TASK | true   | 3                       | 100                           | 0                 | 92                       |               |
+------+-------------+--------------------------+--------+-------------------------+-------------------------------+-------------------+--------------------------+---------------+

执行停写（可选）。

该语句会拦截增删改和部分DistSQL，是可选的。

STOP RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

执行数据一致性校验。

CHECK RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

查询数据一致性校验进度：

SHOW RESHARDING CHECK STATUS 'j51017f973ac82cb1edea4f5238a258c25e89';

示例结果：

+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+
| tables  | result | finished_percentage | remaining_seconds | check_begin_time        | check_end_time          | duration_seconds | error_message |
+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+
| t_order | true   | 100                 | 0                 | 2022-10-25 10:13:33.220 | 2022-10-25 10:13:35.338 | 2                |               |
+---------+--------+---------------------+-------------------+-------------------------+-------------------------+------------------+---------------+

切换元数据。

APPLY RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';

预览生效后的规则：

PREVIEW SELECT * FROM t_order;

示例结果：

+------------------+-----------------------------------------------------------------+
| data_source_name | actual_sql                                                      |
+------------------+-----------------------------------------------------------------+
| ds_0             | select * from v1_t_order_0 UNION ALL select * from v1_t_order_3 |
| ds_1             | select * from v1_t_order_1 UNION ALL select * from v1_t_order_4 |
| ds_2             | select * from v1_t_order_2 UNION ALL select * from v1_t_order_5 |
+------------------+-----------------------------------------------------------------+

恢复停写（可选）。

如果在之前执行了停写，需要进行恢复。

RESTORE RESHARDING SOURCE WRITING 'j51017f973ac82cb1edea4f5238a258c25e89';

完成扩缩容作业。

示例：

COMMIT RESHARDING 'j51017f973ac82cb1edea4f5238a258c25e89';