Logo
Data Encryption

Data Encryption #

Security control has always been an important part of governance, and data security has always been an extremely important and sensitive topic.

Data encryption refers to the transformation of data through encryption rules for certain sensitive information to achieve reliable protection of sensitive and private data. Personal information involving customer security data or some commercially sensitive data, such as ID card number, cell phone number, card number, customer number, etc. needs to be data encrypted according to the relevant access permissions of different teams.

For the need for data encryption, there are generally two situations in real business scenarios:

  1. When a new business is launched, the security team requires that sensitive information involving users, such as bank and cell phone numbers, be encrypted and stored in the database, and then decrypted when in use. Because it is a brand new system, where there is no stock data cleansing problem, it is relatively simple to implement.

  2. In the case of online business, plaintext has been stored in the database before. The department concerned suddenly needs to encrypt the online business for rectification. This scenario generally needs to deal with 3 issues:

    • How to encrypt historical data, i.e. how to clean the data.
    • How to encrypt the new data and store it in the database without changing the business SQL and logic, and then decrypt and retrieve it when it is used.
    • How to securely, seamlessly and transparently migrate business systems between plaintext and ciphertext data.

Overview #

SphereEx-DBPlusEngine provides a complete, secure, transparent, and low-cost data encryption integration solution based on the industry’s demand for encryption and business transformation pain points. Currently, SphereEx-DBPlusEngine has built-in MD5, AES, RC4, SM4, and SM3 algorithms. Supports common database types such as MySQL, PostgreSQL, openGauss, Oracle, etc.

In real business scenarios, the DevOps team often needs to implement and maintain a set of encryption and decryption systems for the needs of the company’s security team. When the encryption scenario changes, the self-maintained encryption system often faces the risk of refactoring or modification.

Additionally, for businesses that are already online, it’s relatively complicated to seamlessly implement encryption transformation in a transparent, secure, and low-risk manner without modifying business logic and SQL. For some non-idempotent encryption algorithms with higher security levels, irreversible idempotent columns are provided for querying, i.e. query auxiliary columns.

To enhance the ease of use of the functionality, SphereEx-DBPlusEngine Data Encryption’s plugin provides cloud-based key management, number sharding, and anti-number sharding capabilities.Supports common database types such as MySQL, PostgreSQL, openGauss, Oracle, etc.

Currently, the database products supported by the SphereEx-DBPlusEngine data encryption plug-in are MySQL, PostgreSQL, openGauss, SQLServer, Oracle, Hive and Presto. The specific supported versions are as follows.

DatabaseVersion supported
1MySQL5.7.x ~ 8.x
2PostgreSQL9.6 and above
3openGauss2.1.0 and above
4SQLServer15.0 and above
5Oracle10g and above
6Hive2.3.2
7Presto0.181, 0.272

Note: MySQL, PostgreSQL, openGauss, and Oracle can return the original data type of the field after decryption. Other database types return character types after decryption.

Basic Concept #

  • Logical column

The logical name used to calculate the encrypted column is the logical identifier of the column in SQL. Logical columns contain ciphertext columns (required), query helper columns (optional), and plaintext columns (optional).

  • dataType

Used to define the type of logical columns, such as INT NOT NULL, VARCHAR(200) DEFAULT NULL, etc. Refer to the official documentation for the definitions of the various dialect fields.

  • cipherColumn

The encrypted data column.。

  • cipherDataType

Used to define the type of the cipher column, Configurable according to the algorithm, if not configured, the default type VARCHAR (4000) is used.

  • assistedQueryColumn

Auxiliary columns for querying. For some non-idempotent encryption algorithms with higher security levels, it provides irreversible idempotent columns for querying.

  • assistedQueryDataType

Used to define the query auxiliary column type, Configurable according to the algorithm, if not configured, the default type VARCHAR (4000) is used.

  • Like Query Columns

For scenarios where a like query is required, this field is used to implement it.

  • Like Query Data Type

Used to define the like query column type, Configurable according to the algorithm, if not configured, the default type VARCHAR (4000) is used.

  • plainColumn

Columns that store plaintext and are used to still provide services during encrypted data migration. It can be deleted after the shuffling of the data is complete.

  • plainDataType

Used to define the plaintext column type, same as the logical column type.

  • Encryption Algorithms

This refers to the specific encryption method used in encrypting data, such as AES and MD5.

  • encrypting

Bulk encryption of unencrypted data in the database.

  • decrypting

Batch decryption of encrypted data in the database.

  • Key Management

How to store and manage keys.

  • Cloud-based Key Management

Since encryption keys are configured in plaintext in the props file as a security risk, SphereEx-DBPlusEngine can manage encrypted keys by adding AWS cloud key management features, for example, using the AWS secretKey feature to save the keys, thus enhancing the security and convenience of the entire encryption. When the program initializes the encryption algorithm, it establishes a connection with AWS to obtain the relevant keys stored in AWS, and then stores the keys in the algorithm. No network interaction with the cloud is involved in the entire data encryption process.

key-on-cloud

  • Encrypting

Currently, SphereEx-DBPlusEngine already provides an encryption solution, for new tables and new businesses, you can directly use encryption rules to configure them, but for existing data tables, you need to shuffle the plaintext fields in these tables and convert them into encrypted content.

SphereEx-DBPlusEngine provides a complete data-cleaning solution. The encrypted scrubbing is triggered by DistSQL. After the program receives the request for the scrubbing task, it will create the scrubbing task based on the current scrubbing rules and encryption rules.

The query task is responsible for querying the user’s table data and getting the plaintext fields that need to be encrypted, and then pushing them to the channel; the update task gets the data from the channel and encrypts the updates. The entire task creation and execution process interacts with the Governance Center, so users can query the progress of the task and clean up the task through the relevant DistSQL.

encrypting

  • Decryption

For a system that is already stored in ciphertext, if you need to restore to the state of plaintext storage, you can use the ability of SphereEx-DBPlusEngine to reverse shuffle and build the corresponding plaintext columns. The operation is similar to shuffling, but the premise of backward data wash is that you need to create the corresponding plaintext column in advance.

  • Re encrypting

For cases such as changing encryption algorithms or changing keys, you can use SphereEx-DBPlusEngine to reshuffle the numbers. The rewashing scheme involves the backwash and shuffle process, which requires manual operation by the user, and plaintext data will be stored during the process.

In the above shuffle process, SphereEx-DBPlusEngine supports the ability of breakpoint continuation to provide a better wash experience for users.

Applicable Scenarios #

  • Data Encryption Scenario for Newly Launched Businesses

For the scenario where you want to quickly launch a new business but need to complete the encryption requirement of the security department, through SphereEx-DBPlusEngine encryption plug-in, you can quickly complete the compliance encryption of data without developing a complex encryption system by yourself. At the same time, the flexibility of SphereEx-DBPlusEngine encryption lock can also help customers avoid the risk of complex reconfiguration and modification caused by the change of encryption scenarios.

    • Data encryption transformation scenarios for existing business

For a mature business that is already online, users need to consider not only the cleaning of historical data but also the switching of old and new functions. With SphereEx-DBPlusEngine encryption plug-in, users can easily complete the encryption transformation of the system, and it can also help users to switch old and new functions safely and quickly, and perform encryption cleaning of the existing data. Users can use the encryption and decryption features transparently without changing any business logic and SQL.

Prerequisites for Use #

SphereEx-DBPlusEngine is designed to make the encryption transformation as seamless as possible for the business. At this stage, some functions are restricted, so during the encryption design stage, we need to investigate whether the business system uses operations such as greater than, less than, etc. See below for the specific usage restrictions of SphereEx-DBPlusEngine.

Restrictions #

  • Need to handle the original stock data in the database by itself.
  • Fuzzy query support %, _, contact escape is not supported for now.
  • Encrypted fields can not support the query case-insensitive function.
  • Encrypted fields can not support comparison operations, such as greater than, less than, ORDER BY, BETWEEN, etc.
  • Encrypted fields can not support calculation operations, such as AVG, SUM, and calculation expressions.
  • Views created on encrypted tables cannot use the encryption and decryption functions.
  • Hive requires the storage format of the encrypted table to be orc, and does not support other storage formats yet.
  • The table involving number shuffling may not have a primary key. If there is a primary key or a unique key, the data type of the primary key or the unique key must be an integer or a string type.

Caution #

Deleting plaintext columns is a high-risk and irreversible operation. Therefore, when the data encryption transformation is completed, it is recommended to run steadily for several months before executing the operation.

Advantage #

  • Automated & transparent data encryption process, users do not need to pay attention to the details of encryption intermediate implementation.
  • Built-in various encryption algorithms, users only need a simple configuration to use.
  • Provides API interface for encryption algorithms, so that users can implement the interface and use custom encryption algorithms for data encryption.
  • For an online business, you can store plaintext data and ciphertext data synchronously, and decide whether to use plaintext or ciphertext columns for query through configuration. It can realize secure and transparent migration of data before and after encryption without changing the business query SQL premise.

Principle Introduction #

SphereEx-DBPlusEngine encrypts the original data by parsing the user input SQL and rewriting the SQL according to the encryption rules provided by the user, and stores both the original data (optional) and the cipher data to the underlying database. When users query the data, it only retrieves the ciphertext data from the database and decrypts it, and finally returns the decrypted original data to the user.

SphereEx-DBPlusEngine automates & transparently encrypts the data, allowing users to use encrypted data as normal data without focusing on the details of data encryption implementation. In addition, SphereEx-DBPlusEngine can provide a relatively complete solution for both existing online businesses for encryption transformation and new online businesses for encryption functions.

principle

SphereEx-DBPlusEngine intercepts the user-initiated SQL, parses and understands the SQL behavior by SQL syntax parser, finds out the fields to be encrypted and the encryption/decryption algorithm to be used to encrypt and decrypt the target fields based on the encryption rules passed in by the user, and then interacts with the underlying database. SphereEx-DBPlusEngine encrypts the plaintext of user requests and stores it in the underlying database; when the user queries, the ciphertext is removed from the database, decrypted and returned to the end user. By blocking the encryption process of data, users do not need to perceive the process of SQL parsing, data encryption and data decryption. They can use encrypted data as if they were using normal data.

For example, if there is a table called t_user in the database, there are two fields pwd_plain for plaintext data, pwd_cipher for ciphertext data, and pwd_assisted_query,for auxiliary query data, and the logicColumn is defined as pwd. Then, the user should write the SQL towards logicColumn, i.e. INSERT INTO t_user SET pwd = '123'. SphereEx-DBPlusEngine receives SQL and finds that pwd is a logicColumn by the encryption configuration provided by the user, so it encrypts the logical column and its The logical column and its corresponding plaintext data are encrypted. SphereEx-DBPlusEngine converts the user-oriented logical column to the underlying database’s plaintext and ciphertext columns by encrypting the column names and data. As the following diagram shows:

encryption-model

That is, based on the encryption rules provided by the user, the user SQL is separated from the underlying data table structure, so that the user SQL writing no longer depends on the real database table structure. The interface, mapping and conversion between the user and the underlying database are handled by SphereEx-DBPlusEngine.

The following picture shows the processing flow and conversion logic when using the encryption module for adding, deleting, and checking, as shown in the figure below.

encryption-workflow

Encryption Algorithm Explanation #

SphereEx-DBPlusEngine provides two types of encryption and decryption interfaces, namely EncryptAlgorithm and QueryAssistedEncryptAlgorithm . On the one hand, SphereEx-DBPlusEngine provides built-in encryption and decryption implementation classes for users, and users only need to configure them. On the other hand, in order to meet the needs of different scenarios, we also open the relevant encryption and decryption interfaces, and users can provide specific implementation classes based on these two types of interfaces. With simple configuration, SphereEx-DBPlusEngine can invoke user-defined encryption and decryption schemes for data encryption.

  • EncryptAlgorithm

This solution encrypts and decrypts the data to be encrypted by providing two methods, encrypt(), decrypt() . When user does INSERT, DELETE, UPDATE, SphereEx-DBPlusEngine will parse, rewrite, route the SQL according to user configuration and call encrypt() to encrypt the data and store it in the database, and when SELECT, it will call decrypt() to decrypt the encrypted data from the database. When SELECT, the decrypt() method is called to reverse decrypt the encrypted data from the database and finally return the original data to the user. Currently, SphereEx-DBPlusEngine provides five specific implementation classes for this type of encryption solution, namely MD5 (irreversible), AES (reversible), RC4 (reversible), SM3 (irreversible) and SM4 (reversible), which can be configured by users to use these five built-in solutions.

  • QueryAssistedEncryptAlgorithm

This scheme is more secure and sophisticated than the first encryption scheme. It is based on the idea that even if the data is the same, such as two users with the same password, they should be stored in the database with different encrypted data. This concept is more conducive to protecting user information and preventing the successful crashing of the database.

It provides three functions for implementation, namely encrypt(), decrypt(), queryAssistedEncrypt(). In the encrypt() phase, the user sets a certain change seed, such as a timestamp. By encrypting the content of the original data + change seed combination, it is guaranteed that the encrypted data will be different even if the original data is the same, due to the presence of the change seed. In decrypt() , the seed data can be decrypted according to the previously specified encryption algorithm.

Although this method can increase the confidentiality of the data, another problem arises: the same data is stored in the database differently, so when the user performs an equivalence query (SELECT FROM table WHERE encryptedColumnn = ?) , it is not possible to query all the same original data.

For this reason, we propose the concept of an auxiliary query column. This auxiliary query column is generated by queryAssistedEncrypt() , which differs from decrypt() in that it encrypts the original data in another way, but the encrypted data produced by this encryption is the same for data with the same original data. The data after queryAssistedEncrypt() is stored into the data to assist in querying the real data. As a result, this additional query column is added to the database table.

Since queryAssistedEncrypt() and encrypt() generate different encrypted data for storage, while decrypt() is reversible, queryAssistedEncrypt() is not. When querying the raw data, we automatically parse, rewrite, and route the SQL, use the auxiliary query column for WHERE conditions, but use decrypt() to encrypt() the encrypted data and return the raw data to the user. This is all transparent to the user.

Currently, SphereEx-DBPlusEngine does not provide a concrete implementation class for this type of encryption solution, but abstracts the idea into an interface for users to implement on their own, and SphereEx-DBPlusEngine will call the concrete implementation class of the solution provided by users to encrypt the data.

Encryption Configuration #

The encryption configuration elements are divided into four parts: data source configuration, encryption algorithm configuration, encryption table configuration and query attribute configuration. The details of which are shown in the following figure:

encrypt-rule

Data source configuration: Refers to the data source configuration.

Encryption algorithm configuration: refers to what encryption algorithm is used for encryption and decryption. Currently SphereEx-DBPlusEngine has five built-in encryption and decryption algorithms: AES, MD5, RC4, SM3 and SM4.

Encryption table configuration: Used to tell SphereEx-DBPlusEngine which column in the data table is used to store cipherColumn, which algorithm to use for encryption (encryptorName), which column is used to store assistedQueryColumn, which algorithm to use for encryption and decryption (assistedQueryEncryptorName), which column is used to store plaintext data (plainColumn), and which column the user wants to use for SQL writing (logicColumn).

We can understand this from the point of view of the existence of the encryption module. The ultimate goal of the encryption module is to shield the underlying encryption of the data, that is, we do not want the user to know how the data is encrypted and decrypted, how the plaintext data is stored in plainColumn and the cipher data is stored in cipherColumn. In other words, we do not want the user to know the existence and use of plainColumn and cipherColumn. exist and are used. Therefore, we need to provide users with a conceptual column that can be separate from the real column of the underlying database, which can be a real column in the database table or not, thus enabling users to change the column names of plainColumn and cipherColumn of the underlying database at will. Or delete the plainColumn and choose to never store plaintext again, only ciphertext. As long as the user’s SQL is written towards this logical column, and the correct mapping between logicColumn and plainColumn and cipherColumn is given in the encryption rules.

Configuration of query property: When both plaintext and ciphertext data are stored in the underlying database table, this property switch is used to decide whether to query the plaintext data in the database table directly and return it, or to query the ciphertext data and return it after decrypting it by SphereEx-DBPlusEngine. This property switch supports field-level, table-level, and global rule-level configuration, and is configured by QUERY_WITH_CIPHER_COLUMN , with field-level priority being the highest.

User Guide #

Encryption - New Launch Business #

The new go-live business is relatively simple because it starts from scratch and there is no historical data cleansing problem. After choosing the appropriate encryption algorithm, such as AES, you only need to configure the logical column (for the user to write SQL) and the cipher column (for the data table to store the cipher data), which can be the same or different. The recommended configuration is as follows (shown in YAML format).

-!ENCRYPT
  encryptors:
    aes_encryptor:
      type: AES
      props:
        aes-key-value: 123456abc
  tables:
    t_user:
      columns:
        pwd:
          cipherColumn: pwd
          encryptorName: aes_encryptor

With this configuration, SphereEx-DBPlusEngine simply converts the logicColumn and cipherColumn , and the underlying data table does not store plaintext, only ciphertext, which is what is required in the security audit section. If you want to store plaintext and ciphertext together in the database, just add plainColumn configuration. The operation process is as follows:

  1. Design the encrypted table and determine the encrypted fields, which can be combined with the table structure, data characteristics and application situation.
  2. Create a logical library in SphereEx-DBPlusEngine and register the storage node (empty).
  3. Create encryption rules, which can be done using SphereEx-Console or DistSQL.
  4. Create the encryption table and complete the encryption table construction.
  5. Depending on the business reality, then choose to keep or delete the plain text fields.

Encrypting - Transformation of Live Businesses #

For the existing business, we need to evaluate the support of SQL after the transformation, and data wash is an essential part.

  1. Confirm the encryption scope Specify the scope of encrypted data according to the compliance requirements.
  2. Confirm the encryption algorithm Select and confirm the suitable encryption algorithm. 3.
  3. Check the business SQL and confirm the support situation Check whether there are non-equivalent cryptographic calculations in the current business, such as like, >, <, etc. If there are, you should consider business transformation.
  4. After completing the above, for projects that are already online, you need to shuffle the stock data. This process does not need to refer to external components, SphereEx-DBPlusEngine can complete the shuffling process, which can be done online.
  5. Enable Cipher Columns When cipher columns are enabled, application requests will interact with the cipher fields through SphereEx-DBPlusEngine, while the plain text fields are still writing data.
  6. When the system has been used and observed for a long enough period of time, the plaintext field can be taken offline. The sensitive fields exist in the system in ciphertext form only.

Decrypting #

  1. Add the corresponding plaintext field
  2. Modify the encryption configuration, add a plaintext field in the configuration, and set the queryWithCipher to true.
  3. Start backwashing
  4. Modify queryWithCipher to false
  5. Delete the encryption rule
  6. Add the encryption field at last

Re Encrypting #

  1. Add the corresponding plaintext field
  2. Start backwashing number
  3. Modify queryWithCipher to false
  4. Add a new encrypted field and clear the original encrypted field
  5. Modify the encryption rules and algorithm
  6. Wash data
  7. Modify queryWithCipher to true
  8. Modify the encryption rule and delete the plaintext field
  9. Delete the original encrypted field and plaintext field

Operation Guide #

Data Encryption - New Business #

  1. Design the encryption table and determine the encryption fields, which can be combined with the table structure, data characteristics and application.
  2. Create a logical library in SphereEx-DBPlusEngine and register the storage node (empty).
  3. Create encryption rules, which can be done using SphereEx-Console or DistSQL.
  4. Create the encryption table and complete the encryption table construction.
  5. Depending on the business reality, then choose to keep or delete the plain text fields.

Encrypting - Mature Business #

  1. Design the encryption table and determine the encrypted fields, which can be combined with the table structure, data characteristics and application.
  2. Create a logical library in SphereEx-DBPlusEngine and register the data source (plaintext data).
  3. Creating encryption rules using DistSQL.
  4. Start shuffling the numbers through the ENCRYPT TABLE command.
  5. Enabling the encrypted fields through the ALTER ENCRYPT RULE command.
  6. Choose to keep or delete the plaintext fields again according to the actual business situation.

Configuration Example #

New online business encryption configuration-DistSQL #

Business scenario analysis: The new online business is relatively simple because everything starts from scratch and there is no historical data cleaning problem.

Solution Description: After choosing a suitable encryption algorithm, such as AES, you only need to configure logical columns (user-oriented SQL writing) and cipher columns (data table storing cipher data), and the logical and cipher columns can be the same or different.

SphereEx-DBPlusEngine only needs to convert logicColumn andcipherColumnassistedQueryColumn , the underlying data table does not store plaintext, only ciphertext, which is the requirement of the security audit part. If you want to store plaintext and ciphertext together in the database, you just need to add plainColumn configuration. The overall processing flow is shown in the following figure.

whole process flow

Environment Description #

InstanceIP AddressService PortHost NameNote
1DBPlusEngine 1.2.0192.168.xx.1023307dbplusengine
2MySQL 5.7.26192.168.xx.1063306ds_encryption

Topology diagram #

Topology

Configuration Process #

  1. Prepare the data source

Create a database named ds_encryption in MySQL.

# mysql -utest -p -h192.168.xx.106

mysql> CREATE DATABASE ds_encryption;
Query OK, 1 row affected (0.05 sec)

mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| ds_encryption      |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)
  1. Create a logical library named testdb in SphereEx-DBPlusEngine.
# mysql -uroot -p -P3307 -h127.0.0.1

mysql> CREATE DATABASE testdb;
Query OK, 0 rows affected (0.85 sec)

mysql> SHOW DATABASES;
+--------------------+
| schema_name        |
+--------------------+
| mysql              |
| information_schema |
| performance_schema |
| sys                |
| testdb             |
+--------------------+
5 rows in set (0.00 sec)
  1. Register the data source to SphereEx-DBPlusEngine
mysql> USE testdb;
Database changed

mysql> REGISTER STORAGE UNIT ds_0 (
    URL="jdbc:mysql://192.168.xx.103:3306/mysql_0?serverTimezone=UTC&useSSL=false",
    USER="test",
    PASSWORD="Test@123"
);
Query OK, 0 rows affected (0.81 sec)
  1. Create sharding rules in SphereEx-DBPlusEngine
mysql> CREATE ENCRYPT RULE t_user (
  COLUMNS(
    (
      NAME = id_card_no,
      PLAIN = id_card_no_plain,
      CIPHER = id_card_no_cipher,
      ENCRYPT_ALGORITHM(
        TYPE(
          NAME = 'AES',
          PROPERTIES('aes-key-value' = '123456abc')
        )
      )
    ),
    (
      NAME = mobile,
      PLAIN = mobile_plain,
      CIPHER = mobile_cipher,
      ENCRYPT_ALGORITHM(
        TYPE(
          NAME = 'AES',
          PROPERTIES('aes-key-value' = '123456abc')
        )
      )
    )
  ),
  QUERY_WITH_CIPHER_COLUMN = true
);
Query OK, 0 rows affected (6.56 sec)

mysql> SHOW ENCRYPT TABLE RULE t_user\G
*************************** 1. row ***************************
                   table: t_user
            logic_column: id_card_no
         logic_data_type:
           cipher_column: id_card_no_cipher
        cipher_data_type:
            plain_column: id_card_no_plain
         plain_data_type:
   assisted_column:
assisted_query_data_type:
       like_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
query_with_cipher_column: true
*************************** 2. row ***************************
                   table: t_user
            logic_column: mobile
         logic_data_type:
           cipher_column: mobile_cipher
        cipher_data_type:
            plain_column: mobile_plain
         plain_data_type:
   assisted_column:
assisted_query_data_type:
       like_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
query_with_cipher_column: true
2 rows in set (0.00 sec)

In the above configuration, the mobile and id_card_no fields in the t_user table are encrypted by AES algorithm.

  1. Create a table and insert several test rows
mysql> CREATE TABLE t_user (
    id INT(8), 
    mobile VARCHAR(50), 
    id_card_no VARCHAR(50)
);
Query OK, 0 rows affected (1.90 sec)

mysql> INSERT INTO t_user (id, mobile, id_card_no)
VALUES (1, 18236483857, 220605194709308170),
       (2, 15686689114, 360222198806088804),
       (3, 14523360225, 411601198601098107),
       (4, 18143924353, 540228199804231247),
       (5, 15523349333, 360924195311103360),
       (6, 13261527931, 513229195302236086),
       (7, 13921892133, 500108194806107214),
       (8, 15993370854, 451322194405305441),
       (9, 18044280924, 411329199808285772),
       (10, 13983621809, 430204195612042092);
Query OK, 10 rows affected (0.61 sec)
  1. Confirm the test data insertion result and execute the join operation for verification.
mysql> SELECT * FROM t_user;
+------+-------------+--------------------+
| id   | mobile      | id_card_no         |
+------+-------------+--------------------+
|    1 | 18236483857 | 220605194709308170 |
|    2 | 15686689114 | 360222198806088804 |
|    3 | 14523360225 | 411601198601098107 |
|    4 | 18143924353 | 540228199804231247 |
|    5 | 15523349333 | 360924195311103360 |
|    6 | 13261527931 | 513229195302236086 |
|    7 | 13921892133 | 500108194806107214 |
|    8 | 15993370854 | 451322194405305441 |
|    9 | 18044280924 | 411329199808285772 |
|   10 | 13983621809 | 430204195612042092 |
+------+-------------+--------------------+
10 rows in set (0.25 sec)

mysql> SELECT * FROM t_user WHERE mobile = '18236483857';
+------+-------------+--------------------+
| id   | mobile      | id_card_no         |
+------+-------------+--------------------+
|    1 | 18236483857 | 220605194709308170 |
+------+-------------+--------------------+
1 row in set (0.50 sec)

mysql>
mysql> SELECT * FROM t_user WHERE mobile = '18236483857'
    -> AND id_card_no = '220605194709308170';
+------+-------------+--------------------+
| id   | mobile      | id_card_no         |
+------+-------------+--------------------+
|    1 | 18236483857 | 220605194709308170 |
+------+-------------+--------------------+
1 row in set (0.26 sec)
  1. Check in MySQL, you can see that the mobile and id_card_no fields in the database both store ciphertext data.
mysql -utest -p -h192.168.xx.103

mysql> USE ds_encryption;
Database changed

mysql> SHOW TABLES;
+-------------------------+
| Tables_in_ds_encryption |
+-------------------------+
| t_user                  |
+-------------------------+
1 row in set (0.00 sec)

mysql> SELECT * FROM t_user;
+------+--------------------------+----------------------------------------------+
| id   | mobile_cipher            | id_card_no_cipher                            |
+------+--------------------------+----------------------------------------------+
|    1 | p31Pkl9nIunYdH+AngyNUA== | pQv0JEkM94QzktJdM8UMg/uLrU71G6n6DALdPp9w6L0= |
|    2 | CV8+uYRaWOzcTQnQX3RcwA== | dCF7k4haK0aIV/d7dtwgzIb4lIFlJ913hrPim1+J278= |
|    3 | jnfu7o44KgN/PV1zhiu7jw== | 8iulp3+XTSv2XHGUUHKV0UsLuFx7yEpQVT+47EFfg94= |
|    4 | ZJDrTv/XIjdqdG1yp0t95w== | iqU6myMGfgI/XnxCtjhbMrwIauriWu8crxPS6BH2pMk= |
|    5 | FnQMYGnFJaiWmTHeNYzbFA== | KAPrCXoo1svMt5NWe0UaKYZIl1rSEVddHbBJO1jPIqw= |
|    6 | lv2ECfTCgQQksvdPp6k3Ug== | BBBPAuwU+iJluI9d9TA+H81BPnVXBaly1BE3EplN4e8= |
|    7 | z46vpnHCFTkIF2EtntxpHQ== | Bc39nPtyz1ji9Rc8k4f7G9CKfPew23mKFwp8guK7ybg= |
|    8 | p/IJdGcCikhpCu5gVZj4jg== | nnv/kS1i7uHXKncUOuLzE8OWM0nGlcGkLokT2dltSaQ= |
|    9 | NvPcQv4w3EqD77+VAX0KCA== | +yeo5LWKNWcekFqYawCKjsctAZqe104DrI7AeZdR/Uk= |
|   10 | xOyg9E0X9lhy9mUx0QyL0A== | U7P1CMcxn6VPHYHPgTAtjHEbb6N6vhGOpdJtVjAdHlA= |
+------+--------------------------+----------------------------------------------+
10 rows in set (0.00 sec)
  1. Delete the plaintext field

(Note: Deleting the plaintext column is a high-risk and irreversible operation, so it is recommended that the operation be performed after several months of stable operation.)

--First delete the inscription field in the database
--Then just refresh the metadata information in the Proxy
mysql> REFRESH table METADATA;

The above is the configuration of the encrypted table using DistSQL, and the following will demonstrate the configuration of encryption through Console.

New Online Business Encryption Configuration-Console #

Environmental Description #

InstanceIP AddressService PortHost NameNote
1DBPlusEngine 1.2.0192.168.xx.1023307dbplusengine
2Console 1.1.0192.168.xx.1018089console
3MySQL 5.7.26192.168.xx.1063306ds_encryption

Topology Diagram #

Topology

Configuration process #

  1. Log in to SphereEx-Console and click the “New Table” button in the “Cluster” - “Objects” page.

According to the reference example, enter the following table creation statement and click “OK”.

CREATE TABLE t_encrypt(
    id INT(8), 
    mobile VARCHAR(50), 
    id_card_no VARCHAR(50)
);
  1. Add encryption plugin

Select the created data table: t_encrypt, click “Add Encryption” -> select “General” encryption -> click “Add Encryption Plugin”

Select “Encryption Algorithm”, select “Columns to be encrypted”, check “Store Plaintext”, click “Add” -> click “OK”.

  1. Insert test data
# mysql -uroot -proot -P3307 -h192.168.xx.102

mysql>use logical001;
mysql> INSERT INTO t_encrypt (id, mobile_plain, id_card_no)
    -> VALUES (1, 18236483857, 220605194709308170),
    ->        (2, 15686689114, 360222198806088804),
    ->        (3, 13983621809, 430204195612042092);
Query OK, 3 rows affected (0.02 sec)
  1. Confirm the data encryption result

Log in to SphereEx-DBPlusEngine to see if the data is displayed in ciphertext.

(Note: If you need plaintext to be displayed in SphereEx-DBPlusEngine, leave “Use encrypted column queries or not” unchecked.)

mysql -uroot -proot -P3307 -h192.168.xx.102

mysql>use logical001;
mysql> SELECT * FROM t_encrypt;
+------+----------------------------------+--------------------+
| id   | mobile                           | id_card_no         |
+------+----------------------------------+--------------------+
|    1 | 18236483857 | 220605194709308170 |                    |
|    2 | 15686689114 | 360222198806088804 |                    |
|    3 | 13983621809 | 430204195612042092 |                    |
+------+----------------------------------+--------------------+
3 rows in set (0.00 sec)

Log in to MySQL to view the data and confirm that it is ciphertext display.

mysql -uroot -proot -P3306 -h192.168.xx.106

mysql> use t_encrypt_db;
mysql> select * from t_encrypt;
+------+----------------------------------+--------------+--------------------+
| id   | mobile_cipher                    | mobile_plain | id_card_no         |
+------+----------------------------------+--------------+--------------------+
|    1 | 6f655ea3b73e32d54ae2d34d672c0c55 | 18236483857  | 220605194709308170 |
|    2 | a2193d0e8de4c2c4f166448e22ba97ae | 15686689114  | 360222198806088804 |
|    3 | 3cf1fd442630071ca664bbd5e683f7f6 | 13983621809  | 430204195612042092 |
+------+----------------------------------+--------------+--------------------+

The above is the process of configuring the encryption table in Console, which is relatively more simplified.

Encryption configuration for live business - Shuffle data #

Shuffle data by table #

For a business system that has been up and running for some time, there will be a large amount of plaintext historical data in the database, and this example will demonstrate how to wash the data into historical items.

  1. Prepare MySQL environment (plaintext) and simulate the live database
mysql> CREATE DATABASE ds_mysql;
Query OK, 1 row affected (0.00 sec)

mysql> USE ds_mysql;
Database changed

mysql> CREATE TABLE t_user (id int NOT NULL PRIMARY KEY, username varchar(50), password varchar(50), email varchar(50));
Query OK, 0 rows affected (0.28 sec)

mysql> INSERT INTO t_user (id, username, password, email) VALUES (1,'a','aa','a@t.c'),(2,'b','bb','b@t.c'),(3,'c','cc','c@t.c');
Query OK, 3 rows affected (0.11 sec)
Records: 3  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+
| id | username | password | email |
+----+----------+----------+-------+
|  1 | a        | aa       | a@t.c |
|  2 | b        | bb       | b@t.c |
|  3 | c        | cc       | c@t.c |
+----+----------+----------+-------+
3 rows in set (0.02 sec)

At this point, the MySQL environment has been prepared and a plaintext table named t_user exists in the ds_mysql database.

  1. Create a logical library named testdb in SphereEx-DBPlusEngine and register the MySQL data source to it
mysql> CREATE DATABASE testdb;
Query OK, 0 rows affected (0.00 sec)

mysql> USE testdb;
Database changed

mysql> REGISTER STORAGE UNIT ds_0 (
    URL="jdbc:mysql://192.168.xx.103:3306/ds_mysql?serverTimezone=UTC&useSSL=false",
    USER="test",
    PASSWORD="Test@123"
);
Query OK, 0 rows affected (0.02 sec)

mysql> SHOW STORAGE UNITS\G
*************************** 1. row ***************************
                           name: ds_0
                           type: MySQL
                           host: 192.168.xx.103
                           port: 3306
                             db: ds_mysql
connection_timeout_milliseconds: 30000
      idle_timeout_milliseconds: 60000
      max_lifetime_milliseconds: 2100000
                  max_pool_size: 50
                  min_pool_size: 1
                      read_only: false
               other_attributes: {"dataSourceProperties":{"cacheServerConfiguration":"true","elideSetAutoCommits":"true","useServerPrepStmts":"true","cachePrepStmts":"true","rewriteBatchedStatements":"true","cacheResultSetMetadata":"false","useLocalSessionState":"true","maintainTimeStats":"false","prepStmtCacheSize":"8192","tinyInt1isBit":"false","prepStmtCacheSqlLimit":"2048","netTimeoutForStreamingResults":"0","zeroDateTimeBehavior":"round"},"healthCheckProperties":{},"initializationFailTimeout":1,"validationTimeout":5000,"keepaliveTime":0,"leakDetectionThreshold":0,"registerMbeans":false,"allowPoolSuspension":false,"autoCommit":true,"isolateInternalQueries":false}
1 row in set (0.00 sec)

mysql> LOAD SINGLE TABLE ds_0.t_user;
Query OK, 0 rows affected (3.87 sec)

mysql> SHOW TABLES;
+------------------+------------+
| Tables_in_testdb | Table_type |
+------------------+------------+
| t_user           | BASE TABLE |
+------------------+------------+
1 row in set (0.01 sec)

The above output confirms that the plaintext table t_user has been identified in the logical library.

  1. Preparing for the wash job

Before we start to shuffle, we need to add the corresponding cipher fields in MySQL. Here we will add 1 cipher fields, for username.

mysql> ALTER TABLE t_user ADD COLUMN username_cipher VARCHAR(50);
Query OK, 0 rows affected (0.33 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> DESC t_user;
+-----------------+-------------+------+-----+---------+-------+
| Field           | Type        | Null | Key | Default | Extra |
+-----------------+-------------+------+-----+---------+-------+
| id              | int(11)     | NO   | PRI | NULL    |       |
| username        | varchar(50) | YES  |     | NULL    |       |
| password        | varchar(50) | YES  |     | NULL    |       |
| email           | varchar(50) | YES  |     | NULL    |       |
| username_cipher | varchar(50) | YES  |     | NULL    |       |
+-----------------+-------------+------+-----+---------+-------+
5 rows in set (0.01 sec)

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+-----------------+
| id | username | password | email | username_cipher |
+----+----------+----------+-------+-----------------+
|  1 | a        | aa       | a@t.c | NULL            |
|  2 | b        | bb       | b@t.c | NULL            |
|  3 | c        | cc       | c@t.c | NULL            |
+----+----------+----------+-------+-----------------+
3 rows in set (0.02 sec)

The current cipher text field content is empty and will be converted to cipher data online by shuffle operation later.

  1. Configure encryption rules in SphereEx-DBPlusEngine
CREATE ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=false)
));
Query OK, 0 rows affected (0.05 sec)


-- NAME:It is the field name in the business. Based on the starting point of application modification-free, it can be set to be consistent with the plain text field name.
-- PLAIN:Plain text field name
-- CIPHER:Cipher text field name
-- ENCRYPT_ALGORITHM:Encryption algorithm related configuration
-- QUERY_WITH_CIPHER_COLUMN:Whether to enable the ciphertext field. The current ciphertext field is empty, so it needs to be set to false.

mysql> SHOW ENCRYPT TABLE RULE t_user\G
*************************** 1. row ***************************
                   table: t_user
            logic_column: username
         logic_data_type:
           cipher_column: username_cipher
        cipher_data_type:
            plain_column: username
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
        query_with_plain: true
1 row in set (0.05 sec)

mysql> PREVIEW SELECT * FROM t_user;
+------------------+----------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                         |
+------------------+----------------------------------------------------------------------------------------------------+
| ds_0             | SELECT t_user.`id`, t_user.`username` AS `username`, t_user.`password`, t_user.`email` FROM t_user |
+------------------+----------------------------------------------------------------------------------------------------+
1 row in set (0.99 sec)

The PREVIEW results show that the request is currently being responded to via plaintext fields, as expected.

  1. Start Encrypting

After configuring the encryption fields and encryption rules, you can start to shuffling data.

CREATE ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=false)
));
Query OK, 0 rows affected (0.05 sec)


-- NAME:It is the field name in the business. Based on the starting point of application modification-free, it can be set to be consistent with the plain text field name.
-- PLAIN:Plain text field name
-- CIPHER:Cipher text field name
-- ENCRYPT_ALGORITHM:Encryption algorithm related configuration
-- QUERY_WITH_CIPHER_COLUMN:Whether to enable the ciphertext field. The current ciphertext field is empty, so it needs to be set to false.

mysql> SHOW ENCRYPT TABLE RULE t_user\G
*************************** 1. row ***************************
                   table: t_user
            logic_column: username
         logic_data_type:
           cipher_column: username_cipher
        cipher_data_type:
            plain_column: username
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
        query_with_plain: true
1 row in set (0.05 sec)

mysql> PREVIEW SELECT * FROM t_user;
+------------------+----------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                         |
+------------------+----------------------------------------------------------------------------------------------------+
| ds_0             | SELECT t_user.`id`, t_user.`username` AS `username`, t_user.`password`, t_user.`email` FROM t_user |
+------------------+----------------------------------------------------------------------------------------------------+
1 row in set (0.99 sec)

The table has both plaintext and secret text, and it is the plaintext field that provides the response for the service at this time.

  1. Enable ciphertext field

When the plaintext and ciphertexts are running in parallel for a period of time, the ciphertext field can be enabled in conjunction with the actual situation.

ALTER ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=true)
));
Query OK, 0 rows affected (0.02 sec)


mysql> SHOW ENCRYPT TABLE RULE t_user\G
*************************** 1. row ***************************
                   table: t_user
            logic_column: username
         logic_data_type:
           cipher_column: username_cipher
        cipher_data_type:
            plain_column: username
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
        query_with_plain: false
1 row in set (0.01 sec)

mysql> PREVIEW SELECT * FROM t_user;
+------------------+-----------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                |
+------------------+-----------------------------------------------------------------------------------------------------------+
| ds_0             | SELECT t_user.`id`, t_user.`username_cipher` AS `username`, t_user.`password`, t_user.`email` FROM t_user |
+------------------+-----------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)

At this point, the business has started using the plaintext field as shown by the PREVIEW output. The wash count operation is completed, and the plaintext field can be changed and adjusted according to the actual usage requirements.

  1. Finish shuffling data
mysql> COMMIT ENCRYPTING j5202p0000e2af8bea809d29f6ab4b3ac4c99b5789;
Query OK, 0 rows affected (0.76 sec)

Shuffle data by column #

After the shuffling is completed, if you need to shuffle additional columns later, you can also shuffle the specified columns. If you need to shuffle the password field this time, the steps are as follows:

Add a new column in MySQL:

mysql> ALTER TABLE t_user ADD COLUMN password_cipher VARCHAR(50);
Query OK, 0 rows affected (0.27 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> DESC t_user;
+-----------------+-------------+------+-----+---------+-------+
| Field           | Type        | Null | Key | Default | Extra |
+-----------------+-------------+------+-----+---------+-------+
| id              | int(11)     | NO   | PRI | NULL    |       |
| username        | varchar(50) | YES  |     | NULL    |       |
| password        | varchar(50) | YES  |     | NULL    |       |
| email           | varchar(50) | YES  |     | NULL    |       |
| username_cipher | varchar(50) | YES  |     | NULL    |       |
| password_cipher | varchar(50) | YES  |     | NULL    |       |
+-----------------+-------------+------+-----+---------+-------+
6 rows in set (0.02 sec)

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+--------------------------+-----------------+
| id | username | password | email | username_cipher          | password_cipher |
+----+----------+----------+-------+--------------------------+-----------------+
|  1 | a        | aa       | a@t.c | T25iRZWea6YFzmT4LqH9ag== | NULL            |
|  2 | b        | bb       | b@t.c | fRwca8JpNUTHo0xBYTfWuw== | NULL            |
|  3 | c        | cc       | c@t.c | GnlS+xZTncXuH5D7JLi4CA== | NULL            |
+----+----------+----------+-------+--------------------------+-----------------+
3 rows in set (0.01 sec)

Modify the encryption rule to query the plaintext column (if the plaintext column has been deleted before, deshuffling needs to be performed first):

mysql> ALTER ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=false)
, (NAME=password, PLAIN=password, CIPHER=password_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=false)
));

mysql> SHOW ENCRYPT TABLE RULE t_user\G
*************************** 1. row ***************************
                   table: t_user
            logic_column: username
         logic_data_type:
           cipher_column: username_cipher
        cipher_data_type:
            plain_column: username
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
        query_with_plain: true
*************************** 2. row ***************************
                   table: t_user
            logic_column: password
         logic_data_type:
           cipher_column: password_cipher
        cipher_data_type:
            plain_column: password
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
        query_with_plain: true
2 rows in set (0.02 sec)

Start shuffling(password column):

mysql> ENCRYPT TABLE t_user (password);
Query OK, 0 rows affected (0.27 sec)

mysql> SHOW ENCRYPTING LIST;
+--------------------------------------------+--------+----------+----------------+--------+---------------------+-----------+
| id                                         | tables | columns  | job_item_count | active | create_time         | stop_time |
+--------------------------------------------+--------+----------+----------------+--------+---------------------+-----------+
| j5202p0000e2af8bea809d29f6ab4b3ac4c99b5789 | t_user | password | 1              | true   | 2023-09-25 17:36:41 | NULL      |
+--------------------------------------------+--------+----------+----------------+--------+---------------------+-----------+
1 row in set (0.03 sec)

mysql> SHOW ENCRYPTING STATUS j5202p0000e2af8bea809d29f6ab4b3ac4c99b5789;
+------+-------------+----------+--------+---------------------+---------------+
| item | data_source | status   | active | finished_percentage | error_message |
+------+-------------+----------+--------+---------------------+---------------+
| 0    | ds_0        | FINISHED | true   | 100                 |               |
+------+-------------+----------+--------+---------------------+---------------+
1 row in set (0.03 sec)

After encryption progress reaches 100%, verify in MySQL:

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+--------------------------+--------------------------+
| id | username | password | email | username_cipher          | password_cipher          |
+----+----------+----------+-------+--------------------------+--------------------------+
|  1 | a        | aa       | a@t.c | T25iRZWea6YFzmT4LqH9ag== | gg9Qid/hjjHgNk8TgeBK5g== |
|  2 | b        | bb       | b@t.c | fRwca8JpNUTHo0xBYTfWuw== | 7i7GQMpZxkWXoXB01NJryQ== |
|  3 | c        | cc       | c@t.c | GnlS+xZTncXuH5D7JLi4CA== | TTDgqe6nMJhnqmy2soADnw== |
+----+----------+----------+-------+--------------------------+--------------------------+
3 rows in set (0.01 sec)

Query the encrypted column instead and verify:

mysql> ALTER ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=true),
(NAME=password, PLAIN=password, CIPHER=password_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=true)
));

mysql> PREVIEW SELECT * FROM t_user;
+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                                     |
+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| ds_0             | SELECT t_user.`id`, t_user.`username_cipher` AS `username`, t_user.`password_cipher` AS `password`, t_user.`email` FROM t_user |
+------------------+--------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+
| id | username | password | email |
+----+----------+----------+-------+
|  1 | a        | aa       | a@t.c |
|  2 | b        | bb       | b@t.c |
|  3 | c        | cc       | c@t.c |
+----+----------+----------+-------+
3 rows in set (0.02 sec)

Finish shuffling:

mysql> COMMIT ENCRYPTING j5202p0000e2af8bea809d29f6ab4b3ac4c99b5789;
Query OK, 0 rows affected (0.69 sec)

Shuffling data by derived column #

After a certain column is shuffled, encryption alone is not enough, such as when fuzzy query is needed. Assume that fuzzy query of username is required. The steps are as follows:

Add a new column in MySQL:

mysql> ALTER TABLE t_user ADD COLUMN username_like VARCHAR(50);
Query OK, 0 rows affected (0.27 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> DESC t_user;
+-----------------+-------------+------+-----+---------+-------+
| Field           | Type        | Null | Key | Default | Extra |
+-----------------+-------------+------+-----+---------+-------+
| id              | int(11)     | NO   | PRI | NULL    |       |
| username        | varchar(50) | YES  |     | NULL    |       |
| password        | varchar(50) | YES  |     | NULL    |       |
| email           | varchar(50) | YES  |     | NULL    |       |
| username_cipher | varchar(50) | YES  |     | NULL    |       |
| password_cipher | varchar(50) | YES  |     | NULL    |       |
| username_like   | varchar(50) | YES  |     | NULL    |       |
+-----------------+-------------+------+-----+---------+-------+
7 rows in set (0.01 sec)

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+--------------------------+--------------------------+---------------+
| id | username | password | email | username_cipher          | password_cipher          | username_like |
+----+----------+----------+-------+--------------------------+--------------------------+---------------+
|  1 | a        | aa       | a@t.c | T25iRZWea6YFzmT4LqH9ag== | gg9Qid/hjjHgNk8TgeBK5g== | NULL          |
|  2 | b        | bb       | b@t.c | fRwca8JpNUTHo0xBYTfWuw== | 7i7GQMpZxkWXoXB01NJryQ== | NULL          |
|  3 | c        | cc       | c@t.c | GnlS+xZTncXuH5D7JLi4CA== | TTDgqe6nMJhnqmy2soADnw== | NULL          |
+----+----------+----------+-------+--------------------------+--------------------------+---------------+
3 rows in set (0.01 sec)

Modify the encryption rule to query the plaintext column (if the plaintext column has been deleted before, you need to perform backwashing first), and add the LIKE_QUERY_COLUMN configuration:

mysql> ALTER ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, LIKE_QUERY_COLUMN=username_like, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), LIKE_QUERY_ALGORITHM(TYPE(NAME='CHAR_DIGEST_LIKE')), QUERY_WITH_CIPHER_COLUMN=false)
, (NAME=password, PLAIN=password, CIPHER=password_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=false)
));

mysql> SHOW ENCRYPT TABLE RULE t_user\G
*************************** 1. row ***************************
                   table: t_user
            logic_column: username
         logic_data_type:
           cipher_column: username_cipher
        cipher_data_type:
            plain_column: username
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column: username_like
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type: CHAR_DIGEST_LIKE
        like_query_props:
        query_with_plain: true
*************************** 2. row ***************************
                   table: t_user
            logic_column: password
         logic_data_type:
           cipher_column: password_cipher
        cipher_data_type:
            plain_column: password
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
        query_with_plain: true
2 rows in set (0.02 sec)

Start shuffling:

mysql> ENCRYPT TABLE t_user (username(LIKE_QUERY));
Query OK, 0 rows affected (0.11 sec)

mysql> SHOW ENCRYPTING LIST;
+--------------------------------------------+--------+----------+----------------+--------+---------------------+-----------+
| id                                         | tables | columns  | job_item_count | active | create_time         | stop_time |
+--------------------------------------------+--------+----------+----------------+--------+---------------------+-----------+
| j5202p0000e2af8bea809d29f6ab4b3ac4c99b5789 | t_user | username | 1              | true   | 2023-09-25 17:47:26 | NULL      |
+--------------------------------------------+--------+----------+----------------+--------+---------------------+-----------+
1 row in set (0.02 sec)

mysql> SHOW ENCRYPTING STATUS j5202p0000e2af8bea809d29f6ab4b3ac4c99b5789;
+------+-------------+----------+--------+---------------------+---------------+
| item | data_source | status   | active | finished_percentage | error_message |
+------+-------------+----------+--------+---------------------+---------------+
| 0    | ds_0        | FINISHED | true   | 100                 |               |
+------+-------------+----------+--------+---------------------+---------------+
1 row in set (0.03 sec)

After encryption progress reaches 100%, verify in MySQL:

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+--------------------------+--------------------------+---------------+
| id | username | password | email | username_cipher          | password_cipher          | username_like |
+----+----------+----------+-------+--------------------------+--------------------------+---------------+
|  1 | a        | aa       | a@t.c | T25iRZWea6YFzmT4LqH9ag== | gg9Qid/hjjHgNk8TgeBK5g== | `             |
|  2 | b        | bb       | b@t.c | fRwca8JpNUTHo0xBYTfWuw== | 7i7GQMpZxkWXoXB01NJryQ== | a             |
|  3 | c        | cc       | c@t.c | GnlS+xZTncXuH5D7JLi4CA== | TTDgqe6nMJhnqmy2soADnw== | d             |
+----+----------+----------+-------+--------------------------+--------------------------+---------------+
3 rows in set (0.00 sec)

Query the encrypted column instead and verify:

mysql> ALTER ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, LIKE_QUERY_COLUMN=username_like, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), LIKE_QUERY_ALGORITHM(TYPE(NAME='CHAR_DIGEST_LIKE')), QUERY_WITH_CIPHER_COLUMN=true)
, (NAME=password, PLAIN=password, CIPHER=password_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=true)
));

mysql> SHOW ENCRYPT TABLE RULE t_user\G
*************************** 1. row ***************************
                   table: t_user
            logic_column: username
         logic_data_type:
           cipher_column: username_cipher
        cipher_data_type:
            plain_column: username
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column: username_like
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type: CHAR_DIGEST_LIKE
        like_query_props:
        query_with_plain: false
*************************** 2. row ***************************
                   table: t_user
            logic_column: password
         logic_data_type:
           cipher_column: password_cipher
        cipher_data_type:
            plain_column: password
         plain_data_type:
   assisted_query_column:
assisted_query_data_type:
       like_query_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
        query_with_plain: false
2 rows in set (0.01 sec)

mysql> PREVIEW SELECT * FROM t_user;
+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                                     |
+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| ds_0             | SELECT t_user.`id`, t_user.`username_cipher` AS `username`, t_user.`password_cipher` AS `password`, t_user.`email` FROM t_user |
+------------------+--------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.04 sec)

mysql> SELECT * FROM t_user;
+----+----------+----------+-------+
| id | username | password | email |
+----+----------+----------+-------+
|  1 | a        | aa       | a@t.c |
|  2 | b        | bb       | b@t.c |
|  3 | c        | cc       | c@t.c |
+----+----------+----------+-------+
3 rows in set (0.03 sec)

Fuzzy Query #

SphereEx-DBPlusEngine can support encrypted columns like query without adding extra storage costs.The following are the steps to verify.

  1. Configure MySQL configuration file

In order to avoid MySQL storing Chinese garbled code which leads to LIKE function error, you need to modify my.cnf in advance to make sure the following configuration is done:

[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
[mysqld]
port=3306
character-set-server=utf8mb4
character-set-filesystem=utf8mb4
collation-server=utf8mb4_general_ci
init-connect='SET NAMES utf8mb4'
  1. Create a logic database and add resources
CREATE DATABASE encrypt_db;

USE encrypt_db;

REGISTER STORAGE UNIT ds_0 (
    HOST="127.0.0.1",
    PORT=3306,
    DB="demo_ds_0",
    USER="root",
    PASSWORD="123456"
), ds_1 (
    HOST="127.0.0.1",
    PORT=3306,
    DB="demo_ds_1",
    USER="root",
    PASSWORD="123456"
);
  1. Create encryption rules
CREATE ENCRYPT KEY MANAGER aws_kms_key_manager (TYPE(NAME='SphereEx:AWS_KMS', PROPERTIES('access-key'='AKIAXRZPPZNGXS6VJUGY','secret-key'='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','aws-region'='ap-northeast-1','secret-name'='dbplus-engine/encrypt-key')));

CREATE ENCRYPT RULE t_user (
        COLUMNS(
                (
                        NAME=user_name,PLAIN=user_name_plain,CIPHER=user_name_cipher,LIKE_COLUMN=user_name_like,
                        ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))),
                        LIKE_QUERY_ALGORITHM(TYPE(NAME='SphereEx:CHAR_TRANSFORM_LIKE',PROPERTIES('key-manager'= "aws_kms_key_manager")))
                ),
                (
                        NAME=password, CIPHER =password_cipher,
                        ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc')))
                ),
                (
                        NAME=email, CIPHER =email_cipher,
                        ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc')))
                ),
                (
                        NAME=telephone,PLAIN=telephone_plain,CIPHER=telephone_cipher,LIKE_COLUMN=telephone_like,
                        ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))),
                        LIKE_QUERY_ALGORITHM(TYPE(NAME='SphereEx:CHAR_TRANSFORM_LIKE',PROPERTIES('key-manager'= "aws_kms_key_manager")))
                )
        ), QUERY_WITH_CIPHER_COLUMN = TRUE
), t_merchant (
        COLUMNS(
                (
                        NAME=business_code,PLAIN=business_code_plain,CIPHER=business_code_cipher,LIKE_COLUMN=business_code_like,
                        ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))),
                        LIKE_QUERY_ALGORITHM(TYPE(NAME='SphereEx:CHAR_TRANSFORM_LIKE',PROPERTIES('key-manager'= "aws_kms_key_manager")))
                ),
                (
                        NAME=telephone,PLAIN=telephone_plain,CIPHER=telephone_cipher,LIKE_COLUMN=telephone_like,
                        ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))),
                        LIKE_QUERY_ALGORITHM(TYPE(NAME='SphereEx:CHAR_TRANSFORM_LIKE',PROPERTIES('key-manager'= "aws_kms_key_manager")))
                )
        ), QUERY_WITH_CIPHER_COLUMN = TRUE
);
  1. Create a logical library and create a test table and insert several test records
USE encrypt_db;

DROP TABLE IF EXISTS t_merchant;


CREATE TABLE t_merchant (
  merchant_id INT PRIMARY KEY, 
  country_id SMALLINT NOT NULL, 
  merchant_name VARCHAR(50) NOT NULL, 
  business_code VARCHAR(50) NOT NULL, 
  telephone CHAR(50) NOT NULL, 
  creation_date DATE NOT NULL);

INSERT INTO t_merchant ......

CREATE TABLE t_user (
  user_id INT PRIMARY KEY, 
  user_name VARCHAR(50) NOT NULL, 
  password VARCHAR(50) NOT NULL, 
  email VARCHAR(50) NOT NULL, 
  telephone CHAR(50) NOT NULL, 
  creation_date DATE NOT NULL);

INSERT INTO t_user ......
  1. Test Number LIKE Query
-- 测试 % 前缀匹配
SELECT * FROM t_merchant WHERE business_code LIKE '%1';
+-------------+------------+---------------+---------------+-------------+---------------+
| merchant_id | country_id | merchant_name | business_code | telephone   | creation_date |
+-------------+------------+---------------+---------------+-------------+---------------+
|           1 |         86 | test101       | 86000001      | 86100000001 | 2017-08-08    |
|          11 |          1 | test111       | 01000011      | 01100000011 | 2017-08-08    |
+-------------+------------+---------------+---------------+-------------+---------------+
2 rows in set (0.01 sec)

-- 测试 % 后缀匹配
SELECT * FROM t_merchant WHERE business_code LIKE '86%';
+-------------+------------+---------------+---------------+-------------+---------------+
| merchant_id | country_id | merchant_name | business_code | telephone   | creation_date |
+-------------+------------+---------------+---------------+-------------+---------------+
|           1 |         86 | test101       | 86000001      | 86100000001 | 2017-08-08    |
|           2 |         86 | test102       | 86000002      | 86100000002 | 2017-08-08    |
|           3 |         86 | test103       | 86000003      | 86100000003 | 2017-08-08    |
|           4 |         86 | test104       | 86000004      | 86100000004 | 2017-08-08    |
|           5 |         86 | test105       | 86000005      | 86100000005 | 2017-08-08    |
|           6 |         86 | test106       | 86000006      | 86100000006 | 2017-08-08    |
|           7 |         86 | test107       | 86000007      | 86100000007 | 2017-08-08    |
|           8 |         86 | test108       | 86000008      | 86100000008 | 2017-08-08    |
|           9 |         86 | test109       | 86000009      | 86100000009 | 2017-08-08    |
|          10 |         86 | test110       | 86000010      | 86100000010 | 2017-08-08    |
+-------------+------------+---------------+---------------+-------------+---------------+
10 rows in set (0.01 sec)
  1. Test letter LIKE query
-- 测试 % 前缀匹配
SELECT * FROM t_user WHERE user_name LIKE '%san';
+---------+-----------+----------+--------------------+-------------+---------------+
| user_id | user_name | password | email              | telephone   | creation_date |
+---------+-----------+----------+--------------------+-------------+---------------+
|      10 | zhangsan  | 111111   | zhangsan@gmail.com | 12345678900 | 2017-08-08    |
|      19 | zhousan   | 123456   | zhousan@gmail.com  | 12345678909 | 2017-08-08    |
+---------+-----------+----------+--------------------+-------------+---------------+
2 rows in set (0.01 sec)

-- 测试 % 后缀匹配
SELECT * FROM t_user WHERE user_name LIKE 'j%';
+---------+-----------+----------+------------------+-------------+---------------+
| user_id | user_name | password | email            | telephone   | creation_date |
+---------+-----------+----------+------------------+-------------+---------------+
|      22 | jerry     | 456789   | jerry@gmail.com  | 12345678912 | 2017-08-08    |
|      23 | james     | 567890   | james@gmail.com  | 12345678913 | 2017-08-08    |
|      27 | jack      | 222333   | jack@gmail.com   | 12345678917 | 2017-08-08    |
|      28 | jordan    | 333444   | jordan@gmail.com | 12345678918 | 2017-08-08    |
|      29 | julie     | 444555   | julie@gmail.com  | 12345678919 | 2017-08-08    |
+---------+-----------+----------+------------------+-------------+---------------+
5 rows in set (0.01 sec)
  1. Test the Chinese character LIKE query
- 测试 % 前缀匹配
SELECT * FROM t_user WHERE user_name LIKE '%靖';
+---------+-----------+----------+-------------------+-------------+---------------+
| user_id | user_name | password | email             | telephone   | creation_date |
+---------+-----------+----------+-------------------+-------------+---------------+
|      34 | 郭靖      | 123123   | guojing@gmail.com | 18012512345 | 2017-08-08    |
+---------+-----------+----------+-------------------+-------------+---------------+
1 row in set (0.01 sec)

-- 测试 % 后缀匹配
SELECT * FROM t_user WHERE user_name LIKE '王%';
+---------+-----------+----------+--------------------------+-------------+---------------+
| user_id | user_name | password | email                    | telephone   | creation_date |
+---------+-----------+----------+--------------------------+-------------+---------------+
|      35 | 王之涣    | 234234   | wangzhihuan@gmail.com | 18012512345 | 2017-08-08    |
|      36 | 王安石    | 345345   | jeff@gmail.com           | 18012512345 | 2017-08-08    |
|      37 | 王羲之    | 345345   | wangxizhi@gmail.com      | 18012512345 | 2017-08-08    |
|      38 | 王莽      | 456456   | wangmang@gmail.com       | 18012512345 | 2017-08-08    |
|      39 | 王勃      | 567567   | wangbo@gmail.com         | 18012512345 | 2017-08-08    |
+---------+-----------+----------+--------------------------+-------------+---------------+
5 rows in set (0.01 sec)

Decrypting #

Assume that encryption or data washing and switching to querying the ciphertext column have been running stably for a period of time, and the plaintext column has been deleted. Example operations:

-- Remove plaintext columns in encryption rules:

mysql> ALTER ENCRYPT RULE t_user (COLUMNS(
(NAME=username, CIPHER=username_cipher, LIKE_QUERY_COLUMN=username_like, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), LIKE_QUERY_ALGORITHM(TYPE(NAME='CHAR_DIGEST_LIKE')), QUERY_WITH_CIPHER_COLUMN=true)
, (NAME=password, CIPHER=password_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=true)
));

-- Remove plaintext columns in MySQL:

mysql> ALTER TABLE t_user DROP COLUMN username;
Query OK, 0 rows affected (0.53 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE t_user DROP COLUMN password;
Query OK, 0 rows affected (0.23 sec)
Records: 0  Duplicates: 0  Warnings: 0

If plaintext columns are needed later, this can be achieved by deshuffling the numbers.

  1. Environment preparation

It’s still the same environment as before.

Add a new plaintext column in MySQL:

mysql> ALTER TABLE t_user ADD COLUMN username VARCHAR(50);
Query OK, 0 rows affected (0.32 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE t_user ADD COLUMN password VARCHAR(50);
Query OK, 0 rows affected (0.23 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> DESC t_user;
+-----------------+-------------+------+-----+---------+-------+
| Field           | Type        | Null | Key | Default | Extra |
+-----------------+-------------+------+-----+---------+-------+
| id              | int(11)     | NO   | PRI | NULL    |       |
| email           | varchar(50) | YES  |     | NULL    |       |
| username_cipher | varchar(50) | YES  |     | NULL    |       |
| password_cipher | varchar(50) | YES  |     | NULL    |       |
| username_like   | varchar(50) | YES  |     | NULL    |       |
| username        | varchar(50) | YES  |     | NULL    |       |
| password        | varchar(50) | YES  |     | NULL    |       |
+-----------------+-------------+------+-----+---------+-------+
7 rows in set (0.02 sec)

mysql> SELECT * FROM t_user;
+----+-------+--------------------------+--------------------------+---------------+----------+----------+
| id | email | username_cipher          | password_cipher          | username_like | username | password |
+----+-------+--------------------------+--------------------------+---------------+----------+----------+
|  1 | a@t.c | T25iRZWea6YFzmT4LqH9ag== | gg9Qid/hjjHgNk8TgeBK5g== | `             | NULL     | NULL     |
|  2 | b@t.c | fRwca8JpNUTHo0xBYTfWuw== | 7i7GQMpZxkWXoXB01NJryQ== | a             | NULL     | NULL     |
|  3 | c@t.c | GnlS+xZTncXuH5D7JLi4CA== | TTDgqe6nMJhnqmy2soADnw== | d             | NULL     | NULL     |
+----+-------+--------------------------+--------------------------+---------------+----------+----------+
3 rows in set (0.01 sec)

Modify the encryption rules and add plaintext column configuration:

mysql> ALTER ENCRYPT RULE t_user (COLUMNS(
(NAME=username, PLAIN=username, CIPHER=username_cipher, LIKE_QUERY_COLUMN=username_like, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), LIKE_QUERY_ALGORITHM(TYPE(NAME='CHAR_DIGEST_LIKE')), QUERY_WITH_CIPHER_COLUMN=true)
, (NAME=password, PLAIN=password, CIPHER=password_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))), QUERY_WITH_CIPHER_COLUMN=true)
));

mysql> SELECT * FROM t_user;
+----+-------+----------+----------+
| id | email | username | password |
+----+-------+----------+----------+
|  1 | a@t.c | a        | aa       |
|  2 | b@t.c | b        | bb       |
|  3 | c@t.c | c        | cc       |
+----+-------+----------+----------+
3 rows in set (0.04 sec)
  1. Create decrypting configuration
mysql> CREATE DECRYPTING PROCESS CONFIGURATION(
READ(
  WORKER_THREAD=20,
  BATCH_SIZE=1000,
  SHARDING_SIZE=1000000
),
WRITE(
  WORKER_THREAD=20,
  BATCH_SIZE=1000
),
STREAM_CHANNEL(TYPE(NAME="MEMORY", PROPERTIES("block-queue-size"=1000)))
);

mysql> SHOW DECRYPTING PROCESS CONFIGURATION;
  1. Start Decrypting
-- Specify any number of deshuffle data fields, that is, deshuffle data by column. If not specified, all encrypted fields will be deshuffle, that is, the data will be deshuffle by tables.mysql> DECRYPT table t_user (username, password);
Query OK, 0 rows affected (1.18 sec)

mysql> SHOW DECRYPTING LIST;
+--------------------------------------------+--------+-------------------+----------------+--------+---------------------+-----------+
| id                                         | tables | columns           | job_item_count | active | create_time         | stop_time |
+--------------------------------------------+--------+-------------------+----------------+--------+---------------------+-----------+
| j5302p0000e2af8bea809d29f6ab4b3ac4c99b5789 | t_user | username,password | 1              | true   | 2023-09-25 18:20:24 | NULL      |
+--------------------------------------------+--------+-------------------+----------------+--------+---------------------+-----------+
1 row in set (0.02 sec)

mysql> SHOW DECRYPTING STATUS j5302p0000e2af8bea809d29f6ab4b3ac4c99b5789;
+------+-------------+----------+--------+---------------------+---------------+
| item | data_source | status   | active | finished_percentage | error_message |
+------+-------------+----------+--------+---------------------+---------------+
| 0    | testdb      | FINISHED | true   | 100                 |               |
+------+-------------+----------+--------+---------------------+---------------+
1 row in set (0.01 sec)
  1. Validate data in MySQL database

After the deshuffling progress reaches 100%, go to MySQL to query and verify:

mysql> SELECT * FROM t_user;
+----+-------+--------------------------+--------------------------+---------------+----------+----------+
| id | email | username_cipher          | password_cipher          | username_like | username | password |
+----+-------+--------------------------+--------------------------+---------------+----------+----------+
|  1 | a@t.c | T25iRZWea6YFzmT4LqH9ag== | gg9Qid/hjjHgNk8TgeBK5g== | `             | a        | aa       |
|  2 | b@t.c | fRwca8JpNUTHo0xBYTfWuw== | 7i7GQMpZxkWXoXB01NJryQ== | a             | b        | bb       |
|  3 | c@t.c | GnlS+xZTncXuH5D7JLi4CA== | TTDgqe6nMJhnqmy2soADnw== | d             | c        | cc       |
+----+-------+--------------------------+--------------------------+---------------+----------+----------+
3 rows in set (0.00 sec)
  1. Finish
mysql> COMMIT DECRYPTING j5302p0000e2af8bea809d29f6ab4b3ac4c99b5789;
Query OK, 0 rows affected (0.60 sec)

Re Encrypting #

  1. Create a database named demo_ds_0in MySQL for preparation
mysql> create datbase demo_ds_0;
  1. Create a logical library in SphereEx-DBPlusEngine and register the data source
mysql> create database encrypt_db;

mysql> use encrypt_db

mysql> REGISTER STORAGE UNIT ds_0 (
    URL="jdbc:mysql://127.0.0.1:3306/demo_ds_0?serverTimezone=UTC&useSSL=false&allowPublicKeyRetrieval=True",
    USER="root",
    PASSWORD="123456",
    PROPERTIES("maximumPoolSize"="100","idleTimeout"="30000")
)
  1. Create encryption rules, create labels and insert data
CREATE ENCRYPT RULE t_encrypt (
COLUMNS(
(NAME=user_id,PLAIN=user_plain,CIPHER=user_cipher,ASSISTED_COLUMN=user_assisted,LIKE_COLUMN=user_like,ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='123456abc'))),ASSISTED_QUERY_ALGORITHM(TYPE(NAME='MD5')), LIKE_QUERY_ALGORITHM(TYPE(NAME='CHAR_DIGEST_LIKE'))),
(NAME=order_id, PLAIN=order_plain,CIPHER =order_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='654321abc'))))
), QUERY_WITH_CIPHER_COLUMN=true);

mysql> create table t_encrypt (
  id int primary key, 
  user_id varchar(200),
  rder_id varchar(200), 
  name varchar(200));

mysql> insert into t_encrypt values 
  (1,'a','aa','aaa'),
  (2,'b','bb','bbb'),
  (3,'c','cc','ccc');
  1. Clear plaintext data in MySQL

mysql> update t_encrypt set user_plain = null,order_plain =null;

mysql> select * from t_encrypt;
+----+--------------------------+----------------------------------+-----------+------------+--------------------------+-------------+------+
| id | user_cipher              | user_assisted                    | user_like | user_plain | order_cipher             | order_plain | name |
+----+--------------------------+----------------------------------+-----------+------------+--------------------------+-------------+------+
|  1 | FrpvlLvZyXewBRRnbeNnbA== | 0cc175b9c0f1b6a831c399e269772661 | `         | NULL       | 7xINTg7CqT86N1xDAHXgxg== | NULL        | aaa  |
|  2 | XWj53fUjwGnpP/LDiJWvow== | 92eb5ffee6ae2fec3ad71c777531578f | a         | NULL       | oTVrU2sLf+HWvYn9FJ04Zw== | NULL        | bbb  |
|  3 | MglbJfLRYx339IXZ3VHDVw== | 4a8a08f09d37b73795649038408b5f33 | d         | NULL       | zsz41oPgr06KG3WWrYixIg== | NULL        | ccc  |
+----+--------------------------+----------------------------------+-----------+------------+--------------------------+-------------+------+
3 rows in set (0.00 sec)
  1. Change key
mysql> REENCRYPT TABLE WITH RULE t_encrypt (
COLUMNS(
(NAME=user_id,PLAIN=user_plain,CIPHER=user_cipher,ASSISTED_COLUMN=user_assisted,LIKE_COLUMN=user_like,ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='665544aaa'))),ASSISTED_QUERY_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='523234abc'))), LIKE_QUERY_ALGORITHM(TYPE(NAME='CHAR_DIGEST_LIKE'))),
(NAME=order_id, PLAIN=order_plain,CIPHER =order_cipher, ENCRYPT_ALGORITHM(TYPE(NAME='AES',PROPERTIES('aes-key-value'='445533ccc'))))
), QUERY_WITH_CIPHER_COLUMN=true);
  • View current re-encrypting jobs
mysql> SHOW REENCRYPTING LIST;

mysql> SHOW REENCRYPTING LIST;
+---------------------------------------+-----------+----------------+--------+---------------------+-----------+
| id                                    | tables    | job_item_count | active | create_time         | stop_time |
+---------------------------------------+-----------+----------------+--------+---------------------+-----------+
| j54014fb26631db0f7149741422d86cc5dfe6 | t_encrypt | 1              | true   | 2022-12-23 18:08:19 | NULL      |
+---------------------------------------+-----------+----------------+--------+---------------------+-----------+
1 row in set (0.21 sec)
  • View re-encrypting job status
-- When the task status is found to reach SUSPEND, the first phase of the task is completed (completion of the backwash count and switch to plaintext query)
mysql> SHOW REENCRYPTING STATUS j54014fb26631db0f7149741422d86cc5dfe6;
+------+-------------+---------+--------+--------------------------------+---------------+
| item | data_source | status  | active | encrypting_finished_percentage | error_message |
+------+-------------+---------+--------+--------------------------------+---------------+
| 0    | encrypt_db  | SUSPEND | true   | 50                             |               |
+------+-------------+---------+--------+--------------------------------+---------------+
1 row in set (0.01 sec)
  • View rule status
SHOW ENCRYPT RULES;
-- Can be seen that the rule uses plaintext queries
mysql> SHOW ENCRYPT RULES\G
*************************** 1. row ***************************
                   table: t_encrypt
            logic_column: user_id
         logic_data_type:
           cipher_column: user_cipher
        cipher_data_type:
            plain_column: user_plain
         plain_data_type:
   assisted_column: user_assisted
assisted_query_data_type:
       like_column: user_like
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=123456abc
     assisted_query_type: MD5
    assisted_query_props:
         like_query_type: CHAR_DIGEST_LIKE
        like_query_props:
query_with_cipher_column: false
*************************** 2. row ***************************
                   table: t_encrypt
            logic_column: order_id
         logic_data_type:
           cipher_column: order_cipher
        cipher_data_type:
            plain_column: order_plain
         plain_data_type:
   assisted_column:
assisted_query_data_type:
       like_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=654321abc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
query_with_cipher_column: false
2 rows in set (0.00 sec)
  1. Confirmation of plaintext data
mysql> select * from t_encrypt;
+----+--------------------------+----------------------------------+-----------+------------+--------------------------+-------------+------+
| id | user_cipher              | user_assisted                    | user_like | user_plain | order_cipher             | order_plain | name |
+----+--------------------------+----------------------------------+-----------+------------+--------------------------+-------------+------+
|  1 | FrpvlLvZyXewBRRnbeNnbA== | 0cc175b9c0f1b6a831c399e269772661 | `         | a          | 7xINTg7CqT86N1xDAHXgxg== | aa          | aaa  |
|  2 | XWj53fUjwGnpP/LDiJWvow== | 92eb5ffee6ae2fec3ad71c777531578f | a         | b          | oTVrU2sLf+HWvYn9FJ04Zw== | bb          | bbb  |
|  3 | MglbJfLRYx339IXZ3VHDVw== | 4a8a08f09d37b73795649038408b5f33 | d         | c          | zsz41oPgr06KG3WWrYixIg== | cc          | ccc  |
+----+--------------------------+----------------------------------+-----------+------------+--------------------------+-------------+------+
3 rows in set (0.01 sec)
  1. Re-encrypting data second stage
START REENCRYPTING j54014fb26631db0f7149741422d86cc5dfe6;
  • View re-encrypting job status
-- When the task status is found to be FINISHED, the second phase of the task is completed (key exchange, clean up encrypted data, shard, and switch to ciphertext query)
    
mysql> SHOW REENCRYPTING STATUS j54014fb26631db0f7149741422d86cc5dfe6;
+------+-------------+----------+--------+--------------------------------+---------------+
| item | data_source | status   | active | encrypting_finished_percentage | error_message |
+------+-------------+----------+--------+--------------------------------+---------------+
| 0    | encrypt_db  | FINISHED | true   | 100                            |               |
+------+-------------+----------+--------+--------------------------------+---------------+
1 row in set (0.00 sec)
  • View current rules
-- The key was found to be replaced and queried using the encrypted column

mysql> show encrypt rules\G
*************************** 1. row ***************************
                   table: t_encrypt
            logic_column: user_id
         logic_data_type:
           cipher_column: user_cipher
        cipher_data_type:
            plain_column: user_plain
         plain_data_type:
   assisted_column: user_assisted
assisted_query_data_type:
       like_column: user_like
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=665544aaa
     assisted_query_type: AES
    assisted_query_props: aes-key-value=523234abc
         like_query_type: CHAR_DIGEST_LIKE
        like_query_props:
query_with_cipher_column: true
*************************** 2. row ***************************
                   table: t_encrypt
            logic_column: order_id
         logic_data_type:
           cipher_column: order_cipher
        cipher_data_type:
            plain_column: order_plain
         plain_data_type:
   assisted_column:
assisted_query_data_type:
       like_column:
    like_query_data_type:
          encryptor_type: AES
         encryptor_props: aes-key-value=445533ccc
     assisted_query_type:
    assisted_query_props:
         like_query_type:
        like_query_props:
query_with_cipher_column: true
2 rows in set (0.01 sec)
  1. Confirm data replacement in MySQL
mysql> select * from t_encrypt;
+----+--------------------------+--------------------------+-----------+------------+--------------------------+-------------+------+
| id | user_cipher              | user_assisted            | user_like | user_plain | order_cipher             | order_plain | name |
+----+--------------------------+--------------------------+-----------+------------+--------------------------+-------------+------+
|  1 | Ktmk9NUTHYXS+OApcRCg+A== | FMAjB5mOAmqotmfX7lQwhQ== | `         | a          | 5owposzhT/Zpk70GAQCBKw== | aa          | aaa  |
|  2 | 9MbqQaWIea72vZomWLoUMQ== | qiZJCKlQF7+/LzjoU9KYTQ== | a         | b          | aHp/Hywm/TkEMgnvjbOLOw== | bb          | bbb  |
|  3 | TLXTYnAbSCstVOz0Cv3iNg== | 3fGlHRInkQn+FcIzMUWS8Q== | d         | c          | 4PhsQk1WvNg1gbD5KfwPpw== | cc          | ccc  |
+----+--------------------------+--------------------------+-----------+------------+--------------------------+-------------+------+
3 rows in set (0.00 sec)
  1. Commit re-encrypting job
mysql> COMMIT REENCRYPTING j54014fb26631db0f7149741422d86cc5dfe6;

Index support instructions after table encryption #

  1. The efficiency of encrypted column indexes needs to be measured based on the encryption algorithm.
  2. Homomorphism, type-preserving, and order-preserving algorithms are currently under development, and homomorphism and order-preserving algorithms can be quickly implemented.

Detailed description

FAQ #

  1. Is it possible to do the wash operation online? Does it have any impact on performance? Yes, SphereEx-DBPlusEngine provides threaded configuration methods to minimize the performance impact on business. At the same time, it is recommended that data shuffling operations should be performed in downtime as much as possible for better performance.
  2. Is it possible to continue shuffling data if the shuffle operation is unexpectedly interrupted? Yes. You can re-execute the shuffle command to continue the operation.