Shadow DB #
Background #
Under the distributed application architecture based on microservices, businesses require multiple services to be completed through a series of services and middleware calls. The pressure testing of a single service can no longer reflect the real scenario.
In the test environment, the cost of rebuild complete set of pressure test environment similar to the production environment is too high. It is usually impossible to simulate the complexity and data of the production environment.
So, it is the better way to use the production environment for pressure test. The test results obtained real capacity and performance of the system accurately.
Challenges #
Pressure testing on production environment is a complex and huge task. Coordination and adjustments between microservices and middlewares required to cope with the transparent transmission of different flow rates and pressure test tags. Usually we will build a complete set of pressure testing platform for different test plans.
Data isolation have to be done at the database-level, in order to ensure the reliability and integrity of the production data, data generated by pressure testing routed to test database. Prevent test data from polluting the real data in the production database.
This requires business applications to perform data classification based on the transparently transmitted pressure test identification before executing SQL, and route the corresponding SQL to the corresponding data source.
Goal #
DBPlusEngine focuses on data solutions in pressure testing on production environment.
The main goal of the DBPlusEngine shadow Database module is routing pressure testing data to user defined database automatically.
Core Concept #
Production Database #
The database used for production data.
Shadow Database #
The database for pressure testing data isolation.
Shadow Algorithm #
The shadow algorithms are closely related to business, 2 types of shadow algorithms provided:
- Column based shadow algorithm
Recognize data from SQL and route to shadow databases. Suitable for test data driven scenario.
- Hint based shadow algorithm
Recognize comment from SQL and route to shadow databases. Suitable for identify passed by upstream system scenario.
Usage Norms #
Supported #
- Hint based shadow algorithm support all SQL.
- Column based shadow algorithm support part of SQL.
Unsupported #
Hint based shadow algorithm #
- None
Column based shadow algorithm #
- Does not support DDL.
- Does not support range, group and subquery, for example: BETWEEN, GROUP BY … HAVING…;
SQL support list:
- INSERT
| SQL | Supported |
|---|---|
| INSERT INTO table (column,…) VALUES (value,…) | Y |
| INSERT INTO table (column,…) VALUES (value,…),(value,…),… | Y |
| INSERT INTO table (column,…) SELECT column1 from table1 where column1 = value1 | N |
- SELECT/UPDATE/DELETE
| Condition | SQL | Supported |
|---|---|---|
| = | SELECT/UPDATE/DELETE … WHERE column = value | Y |
| LIKE/NOT LIKE | SELECT/UPDATE/DELETE … WHERE column LIKE/NOT LIKE value | Y |
| IN/NOT IN | SELECT/UPDATE/DELETE … WHERE column IN/NOT IN (value1,value2,…) | Y |
| BETWEEN | SELECT/UPDATE/DELETE … WHERE column BETWEEN value1 AND value2 | N |
| GROUP BY … HAVING… | SELECT/UPDATE/DELETE … WHERE … GROUP BY column HAVING column > value | N |
| Subquery | SELECT/UPDATE/DELETE … WHERE column = (SELECT column FROM table WHERE column = value) | N |