Case study: solving a fintech user’s data storage and expansion with hundreds of billions of rows #
Customer Pain Points #
To cope with the rapid development of business and the surge of massive amounts of data, data architectures have evolved several times. Nevertheless, with the product upgrade iteration, the earlier solution became an immediate problem. The data sharding scheme achieved through the business framework led to the increase of business code complexity, rising maintenance costs and the disadvantages of tight coupling. Each application upgrade requires more energy to accordingly adjust the sharding, making it is difficult for R & D teams to focus on the business itself. After lengthy consideration, the technical team began to consider using mature sub database and sub table components to undertake this part of the work, so that the business system upgrade and architecture adjustment are no longer complex.
Customer Research #
The comparison between ShardingSphere based sharding and self-developed framework based sharding is as follows:
| sharding based on self-developed frame | sharding based on DBPlusEngine | |
|---|---|---|
| Performance | High | High |
| Code coupling | High | Low |
| Business intrusion degree | High | Low |
| Upgrade difficulty | High | Low |
| Expansibility | Commonly | Good |
Advantages of DBPlusEngine:
- Mature and stable products
- Ultimate performance
- Processing massive data
- Flexible extension of Architecture
Customer Transformation and Architecture #
SphereEx-DBPlusEngine has provided more support and improvement to the function and performance of the product in the process of landing the Internet financial business of the main customers, and the product has experienced the polishing of typical cases again.
Upgrade SQL engine
Distributed primary key
Business sharding key value injection
SQL parsing result cache
JDBC metadata information cache
Use of bind table and broadcast table
Automatic execution engine and stream merging
Through the cooperation of the two teams, the indicators of the combination of the customer’s Internet financial business and SphereEx-DBPlusEngine meet the expectations, and the performance is almost consistent with that of native JDBC.
The reconstructed structure is as follows:
