A Fintech User’s Data Storage and Expansion with Hundreds of Billions of Data Rows Solved #
Customer Pain Points #
To cope with the rapid development of business and the surge of massive amounts of data, data architectures have evolved several times. Nevertheless, with product upgrade iteration, the earlier solution became an immediate problem.
The data-sharding scheme achieved through the business framework led to an increase in business code complexity, rising maintenance costs, and the disadvantages of tight coupling. Each application upgrade required more energy to adjust the sharding accordingly, making it difficult for R&D teams to focus on the business itself.
After lengthy consideration, the technical team began to consider using mature sub-database and sub-table components to undertake this part of the work, so that the business system upgrade and architecture adjustment were no longer complex.
Customer Research #
The comparison between DBPlusEngine-based sharding and self-developed framework-based sharding is as follows:
| Sharding based on the self-developed framework | Sharding based on DBPlusEngine | |
|---|---|---|
| Performance | High | High |
| Code coupling | High | Low |
| Business intrusion degree | High | Low |
| Upgrade difficulty | High | Low |
| Expandability | Commonly | Good |
Advantages of DBPlusEngine:
- Mature and stable product
- Ultimate performance
- Processing massive amounts of data
- Flexible architecture extension
Customer Transformation and Architecture #
SphereEx-DBPlusEngine has provided more support and improvement to the function and performance of the product in the process of landing the Internet financial business of the main customers, and the product has experienced the polishing of typical cases again.
Upgrade SQL engine
Distributed primary key
Business sharding key value injection
SQL parsing result cache
JDBC metadata information cache
Use of bind table and broadcast table
Automatic execution engine and stream merging
Through the cooperation of the two teams, the indicators of the combination of the customer’s Internet financial business and SphereEx-DBPlusEngine meet the expectations, and the performance is almost consistent with that of native JDBC. The proposed architecture is illustrated in the following figure:
