Cluster Installation & Uninstallation #

Prerequisite

Before creating a cluster, the creation or registration of related resources (host, database) and components (governance centre, monitoring centre, log centre) need to be completed.

Driver Description

When SphereEx-Boot installs the cluster, the user needs to download the database driver and put it in the proxy directory.

When SphereEx-Console installs the cluster, it comes with MySQL 8.0 driver.

SphereEx-Boot Cluster Installation #

Description: Installing and uninstalling a cluster is actually installing and uninstalling computing nodes. Creating a cluster with SphereEx-Boot will create a management centre simultaneously.

Link to [Create a cluster with SphereEx-Boot] in the Quick Start section

SphereEx-Boot Cluster Uninstallation #

Uninstalling the cluster using SphereEx-Boot will uninstall the Management Center synchronously.

Use the command spex cluster uninstall --name <cluster-name> to uninstall the cluster, which will delete the deployment directory.

SphereEx-Console Cluster Installation #

Applicable Scenario

Installing computing nodes and registering storage nodes the governance centre has created or registered is necessary for creating a cluster.

Precautions

• The host has been registered, and the account of the registered host user has read and write permissions to the directory.

• Database instance has been registered.

• When installing and starting the compute node, Engine-Agent will be started at the same time by default.

Operation Steps

1. Login SphereEx-Console
2. Click Cluster management->Cluster enter cluster list.
3. Click the New button in the upper right corner.
4. Fill in the information, see the table below for specific parameters.

Cluster Information

Compute Node

Storage Node（Optional）

5. Click OK to complete the cluster installation.

SphereEx-Console Register Cluster #

Applicable Scenario

Register the installed cluster to SphereEx-Console.

Precautions

The cluster has been installed and can provide services normally. If there is an Oracle storage unit in the registered cluster, the proxy_frontend_database_protocol_type parameter needs to adjust to MySQL, put the Oracle driver into the lib directory of the proxy, and restart the operation in the Console

Operation Steps

1. Login SphereEx-Console。
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click the Register button in the upper right corner to enter the registration page.
4. Fill in the information, see the table below for specific parameters.

Basic Information

Compute Node Information

5. Click Register Host to complete the host registration, and fill in the information in the registration section of the reference host.

6. Click OK to complete the cluster registration.

SphereEx-Console Cluster Uninstallation #

Applicable Scenario

Delete the cluster, uninstall the compute node software from SphereEx-Console.

Precautions

If the cluster is stopped, check Uninstall the software synchronously to uninstall the computing node software synchronously, and uninstall the monitoring and log configuration, delete the namespace information, and SphereEx-Console releases the management of the cluster.

Operation Steps

1. Login SphereEx-Console.
2. Select Cluster Management > Cluster in the upper left corner of the console to enter the cluster list.
3. Click the Delete button in the Action column of the list.
4. Click OK in the prompt box, and check Uninstall Software Synchronously to complete the uninstallation of the cluster.

SphereEx-Console Unregister Cluster #

Applicable Scenario

Unmanage the cluster from SphereEx-Console.

Precautions

It’s not required to stop the cluster, only to unmanage the cluster.

Operation Steps

1. Login SphereEx-Console.
2. Select Cluster Management > Cluster in the upper left corner of the console.
3. Click the Delete button in the Action column of the list.
4. Click Confirm in the prompt box, do not check Synchronous Uninstall Software to complete the logout of the cluster.

Cluster Operation And Maintenance Management #

Description Of Cluster Management Functions #

Specify Cluster Refresh #

Applicable Scenario

Specify cluster refresh

Prerequisite

The cluster has already been registered or installed.

Operation Steps

1. Login SphereEx-Console.
2. Select Cluster Management > Cluster in the upper left corner of the console to enter the cluster list.
3. Click the Refresh button in the List Actions column to complete the cluster refresh.

All Cluster Refresh #

Applicable Scenario

Refresh all clusters in the list

Prerequisite

The cluster has already been registered or installed.

Operation Steps

1. Login SphereEx-Console.
2. Select Cluster Management > Cluster in the upper left corner of the console to enter the cluster list.
3. Click the Refresh button in the upper right corner to complete the refresh.

Parameter Management #

Applicable Scenario

Manage cluster parameters, modify parameters within the value range, and reset parameters to default values.

Precautions

The cluster can access

Operation Steps

1. Login SphereEx-Console.
2. In the upper left corner of the console, select Cluster Management > Cluster to enter the cluster list.
3. Click the parameter management button in the list operation column to enter the parameter management page.
4. Click the reset button to pop up the parameter reset page. The lower left corner will prompt whether the modification will take effect immediately. Click OK to reset the parameter to the default value.
5. Click the Modify button to pop up the parameter modification page. The lower left corner prompts whether the modification will take effect immediately. Click OK to complete the parameter modification.

License #

Applicable Scenario

Manage cluster license and update license

Precautions

The cluster can access

Operation Steps

1. Log in to SphereEx-Console.
2. In the upper left corner of the console, select Cluster Management > Cluster to enter the cluster list.
3. Click the License button in the list operation column to enter the management page.

When there is a license, the basic information of the license is visible.

When there is no license, no license will prompt.

4. Click the back button in the upper left corner to return to the cluster list.
5. Click Update License in the upper right corner. Add a License key, click Preview to preview the License information.
6. Click OK to complete the license update

Modify #

Applicable Scenario

You can modify the cluster description and SphereEx-Console login computing node information.

Prerequisite

The cluster has already been registered or installed.

Operation Steps

1. Log in to SphereEx-Console.
2. In the upper left corner of the console, select Cluster Management > Cluster to enter the cluster list.
3. Click the Modify button in the list operation column to enter the modification page.
4. Fill in the following information:

5. Click Confirm in the prompt box to complete the modification.

Start #

Applicable Scenario

Start specified cluster

Precautions

It cannot be restarted repeatedly. If it has already been started, it will prompt that it has been started.

Cluster startup sequence:

a. Governance Center: It is a necessary condition for the cluster to start, and monitors whether it is started; if it is not started, the user is prompted to start it manually, and the cluster will fail to start.

b. Log Center: It is an optional condition for cluster startup to monitor whether it is started; if there is an exception, an error will be prompted, and if it is not started, the user will be prompted to start it manually, which will not affect the startup of the cluster.

c. Monitoring Center: It is an optional condition for cluster startup to monitor whether it is started; if there is an exception, an error will be prompted, and if it is not started, the user will be prompted to start it manually, which will not affect the startup of the cluster.

d. Computing node: It is a mandatory condition for cluster startup to monitor whether it is started; if there is an exception, an error will be prompted, and if it is not started, the user will be prompted to start it manually; if some nodes are available, they can be started, at least one.

e. Storage node: If there is a storage node, it is required. If there is an exception, the cluster will fail to start. If there is no storage node, the cluster startup will not be affected.

Operation Steps

1. Log in to SphereEx-Console.
2. In the upper left corner of the console, select Cluster Management > Cluster to enter the cluster list.
3. Click the Launch button in the Action column of the list to complete the launch.

Stop #

Applicable Scenario

Stop the specified cluster

Precautions

• Stopping the cluster actually only stops the computing nodes. After the cluster is stopped, it will be unusable. Please operate with caution.
• After the cluster is stopped, the button in the operation column can be clicked, but there is no data.

Operation Steps

1. Log in to SphereEx-Console.
2. In the upper left corner of the console, select Cluster Management > Cluster to enter the cluster list.
3. Click the Stop button in the list action column to finish stopping.

Config Logs #

Applicable Scenario

Centrally manage the logs of cluster computing nodes on the SphereEx-Console platform.

Configure the logs of the cluster, that is, configure the logs of the computing nodes, and aggregate the running logs and slow logs to the log center. You can filter, view, and export logs through the slow log and log functions in the SQL management of the Console. Among them, you can directly view the execution plan of SQL in the slow log.

Precautions

• the cluster has been installed or registered
• The host where the cluster computing node is located has been registered, and the account of the registered host user has read and write permissions for the directory
• Log Center has been installed

Operation Steps

1. Log in to SphereEx-Console.
2. In the upper left corner of the console, select Cluster Management > Cluster to enter the cluster list.
3. Click the Configure Log button in the list operation column to enter the configure log page.
4. Fill in the following information:

5. Click Register Host to complete the host registration, and fill in the information in the registration part of the reference host.
6. Click OK in the prompt box to complete the configuration.

Config Monitoring #

Please refer [cluster monitoring](/docs/operation-maintenance/9-monitor-alarm/#Cluster Monitoring)

Compute node management function description #

Add Node #

Applicable Scenario

Expand computing resources for the cluster

Precautions

The host has been registered, and the account of the registered host user has read and write permissions to the directory.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click Add Node in the upper right corner to enter the Add Node page.
5. Fill in the information, the specific parameters are shown in the table below:

6. Click OK to complete adding compute nodes.

Probe #

Applicable Scenario

Probe Compute Node

Precautions

The cluster has already been registered or installed

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click the Detection button in the list operation column to complete the node detection.

Refresh #

Applicable Scenario

Refresh compute node

Precautions

The cluster has already been registered or installed.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click the Refresh button in the list operation column to complete the node refresh.

Enable #

Applicable Scenario

Enable disabled nodes

Precautions

Node is disabled

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click the Enable button in the list operation column to complete the node activation.

Disable #

Applicable Scenario

Disable the node, the disabled node will not have new traffic, and the process being executed will not be affected.

Precautions

Node is enabled.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click the Disable button in the list operation column to complete the node disabling.

Use SphereEx-Console to start computing nodes #

Applicable Scenario

Start a stopped node

Precautions

• the node is stopped

• When the cluster is stopped, you can click the management node to start the computing nodes one by one. As long as one computing node is available, the cluster will be available.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click the Start button in the list operation column to complete the node startup.

Start the compute nodes individually #

1. startup steps

  a. Download the latest release of DBPlusEngine-Proxy.

  b. After decompression, modify conf/server.yaml and files starting with config- prefix, such as: conf/config-xxx.yaml file, and configure sharding rules and read-write separation rules.

  c. Please run bin/start.sh for Linux operating system, and bin/start.bat for Windows operating system to start ShardingSphere-Proxy.

2. Choose a database protocol

Using PostgreSQL

  a. Use any PostgreSQL client connection. Such as: psql -U root -h 127.0.0.1 -p 3307

Using MySQL

  b. Copy the JDBC driver for MySQL to the directory ext-lib/.

  c. Use any MySQL client connection. Such as: mysql -u root -h 127.0.0.1 -P 3307

Using openGauss

  a. Copy the openGauss JDBC driver prefixed with the org.opengauss package name to the directory ext-lib/.

  b. Use any openGauss client connection. Such as: gsql -U root -h 127.0.0.1 -p 3307

3. Select metadata persistence repository

Using ZooKeeper

The ZooKeeper Curator client is integrated by default.

Using Etcd

Copy Etcd’s client driver to the directory ext-lib/.

4. Use distributed transactions

For details, please refer to Distributed Transactions.

5. Use a custom algorithm

When users need to use a custom algorithm class, they can configure and use the custom algorithm in the following ways, taking sharding as an example:

  a. Implement the algorithm implementation class defined by the ShardingAlgorithm interface.

  b. Create a META-INF/services directory under the project resources directory.

  c. Create a new file org.apache.shardingsphere.sharding.spi.ShardingAlgorithm in the META-INF/services directory

  d. Write the absolute path of the implementation class to the file org.apache.shardingsphere.sharding.spi.ShardingAlgorithm

  e. Pack the above Java files into a jar package.

  f. Copy the above jar package to the ext-lib/ directory decompressed by DBPlusEngine-Proxy.

  g. Configure the Java file reference of the above-mentioned custom algorithm implementation class in the YAML file. For details, please refer to the configuration rules.

6. Precautions

  a. DBPlusEngine-Proxy uses port 3307 by default, and you can add parameters as the startup port number through the startup script. Such as: bin/start.sh 3308

  b. DBPlusEngine-Proxy uses conf/server.yaml to configure registry, authentication information and public properties.

  c. DBPlusEngine-Proxy supports multiple logical data sources, and each YAML configuration file named with the config- prefix is a logical data source.

  d. The default listening address of DBPlusEngine-Proxy is 0.0.0.0, and the listening address can be specified through the startup script, which can be obtained through bin/start.sh –help.

Use SpehreEx-Console to stop computing nodes #

Applicable Scenario

Stop the node in the starting state

Precautions

• the node is started
• The cluster will stop after the last computing node is stopped

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click the Stop button in the list operation column to complete the node stop.

Stop compute nodes individually #

For Linux operating system, please run bin/stop.sh, and for Windows operating system, please run bin/stop.bat to start ShardingSphere-Proxy.

Delete #

Applicable Scenario

compute nodes from the cluster

Precautions

Checking the Synchronous Uninstall Software will synchronously uninstall the software of the computing nodes, and uninstall the monitoring and log configuration

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column to enter the calculation point list by default.
4. Click the Delete button in the Action column of the list.
5. Click Confirm in the prompt box to delete the node.

Storage node management function description #

Register #

Applicable Scenario

Cluster expansion storage nodes

Precautions

• The database instance has already been registered.
• The newly added storage node belongs to the cluster and is not used.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column, click Storage Node to enter the list of storage points.
4. Click the Add Node button in the upper right corner to enter the Add Node page.
5. Fill in the information, see the table below for the specific parameters.

6. Click OK to complete the node addition.

Probe #

Applicable Scenario

Probe Storage Node

Precautions

The cluster has already been registered or installed

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column, click Storage Node to enter the list of storage points.
4. Click the Detection button in the list operation column to complete the node detection.

Refresh #

Applicable Scenario

Refresh storage node

Precautions

The cluster has already been registered or installed

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column, click Storage Node to enter the list of storage points.
4. Click the Refresh button in the list operation column to complete the node refresh.

Modify #

Applicable Scenario

Modify storage node information

Precautions

Only the user name and password of the storage node can be modified, which is applicable to the scenario where the user name and password of the user storage node are changed. Modifying the user name and password of the storage node will not affect the instance to which it belongs.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column, click Storage Node to enter the list of storage points.
4. Click the Modify button in the list operation column to enter the page for modifying storage nodes.
5. Fill in the information, see the table below for the specific parameters.

Delete #

Applicable Scenario

Delete unused storage nodes

Precautions

storage node is not used by logical database

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Cluster to enter the cluster list.
3. Click Manage Node in the operation column, click Storage Node to enter the list of storage points.
4. Click the Delete button in the list operation column to complete the deletion of the storage node.

Logical Database Management #

List of logical database management functions

New #

Applicable Scenario

Create new logical database

Precautions

• The cluster can be accessed normally, and there are storage nodes available in the cluster
• A storage node that is not used by any logic database is an available storage node

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Logical Database to enter the list of logical databases.
3. Click the New button in the upper right corner to enter the page for creating a new logic database.
4. Fill in the information, see the table below for specific parameters.

5. Click OK to complete the creation of the logic database.

Refresh #

Applicable Scenario

Refresh logical database

Precautions

The cluster can be accessed normally

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Logical database to enter the list of logical databases.
3. Click the Refresh button in the upper right corner to complete the logic database refresh.

Slow log #

Applicable Scenario

View the slow log generated by the logic database

Precautions

• The cluster can be accessed normally
• slow log is turned on
• Slow logs can be filtered by SQL statement, DistSQL user, computing node, fuzzy query; time, SQL type, execution time and equivalent query

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Logical database to enter the list of logical databases.
3. Click the Slow Log button in the operation column to jump to the slow log page.

The content of the slow log includes: time, user, computing node, logic database, SQL type, execution time, and SQL statement.

Hover to View Execution Plan to jump to the execution plan page and view the execution path of the slow log.

4. Click Export to export all logs that meet the criteria.

Storage Node #

Applicable Scenario

Manage Storage Nodes for logical databases

Precautions

• The cluster can be accessed normally
• Storage nodes not used by logical tables can be removed

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Logical Database to enter the list of logical databases.
3. Click the Storage Node button in the operation column to jump to the storage node management page.
4. Complete the adjustment of the database storage node by adjusting the storage node in the Optional column and Selected column.

UDF #

Applicable Scenario

Use udf to encrypt stored procedures of oracle database

Precautions

• The cluster can be accessed normally.
• There is only one logical database, and one storage unit is registered in the logical database. The storage unit type is oracle.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Logical Database to enter the list of logical databases.
3. Click operation column UDF button，jump to the UDF configuration page.
4. Turn UDF on or off by adjusting the click switch.

For a detailed introduction to the stored procedure function of Oracle database, please refer to Oracle Stored Procedure Encryption

Delete #

Applicable Scenario

Delete logical database

Precautions

• The cluster can be accessed normally.
• Deleting a database from a cluster does not delete the cluster’s storage nodes.
• Deleting the logic database will delete the plug-in configuration in the logic database synchronously, and it cannot be restored. Please operate with caution.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Logical Database to enter the list of logical databases.
3. Click the Delete button in the action column.
4. Click the Confirm button in the prompt to complete the deletion of the logic database.

Table #

Create New Table #

Applicable Scenario

Create a new table in the logic database

Precautions

• The cluster can be accessed normally, and there is a logical database
• When the logical database is registered in the Oracle storage unit, the semicolon “;” cannot be included in the table creation statement.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the New Table button in the upper right corner to enter the page for creating a new table. When creating a new table, the sharding plug-in and encryption plug-in are optional; the sharding plug-in and encryption plug-in can be created in conventional and custom ways respectively.
4. Fill in the information, see the table below for specific parameters.

table configuration

Add sharding configuration General configuration

Sharding configuration: regular means formatted definition rules; custom means use DistSQL to define rules, and the regular way does not need to define storage nodes; choose one of the regular and custom configurations, and the corresponding configuration item will appear in whichever way you choose.

Add sharding configuration Custom configuration

Add encryption configuration General configuration

Encryption plug-in configuration: regular means formatted definition rules; custom means use DistSQL to define rules, choose one of the regular and custom configurations, and the corresponding configuration items will appear in whichever way you choose.

[Use Encrypted Column Query] Check it, use the encrypted data column when querying; uncheck, use the unencrypted data column when querying.

Checked by default.

When the plaintext storage is unchecked, [Use Encrypted Column Query] must be checked and cannot be changed.

When all plaintext storage is checked, [Use Encrypted Column Query] is allowed to be changed to unchecked.

Add encryption configuration Custom configuration

Add masking configuration General configuration

Masking configuration：regular means formatted definition rules; custom means use DistSQL to define rules, and the regular way does not need to define storage nodes; choose one of the regular and custom configurations, and the corresponding configuration item will appear in whichever way you choose.

Add Masking configuration Custom configuration

5. Click OK to finish creating the table.

Batch Encryption and Masking #

Applicable Scenario

Configure encryption, masking, or encryption and masking in batches based on reports generated by the Sensitive Data Identification feature.

Precautions

The cluster can be accessed normally, there is a logical database, and a report corresponding to the storage unit of the logical database has been generated in the sensitive data identification function.

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Table to enter the table list.
3. Click the Batch Encryption and Masking button in the upper right corner to enter the table group management page.
4. Select report
5. Adjust matching (optional)
6. Add masking or encryption configurations for columns not included in reports
7. Delete the masking or encryption configuration of the selected column
8. Click the edit button in the encryption or masking configuration column to modify the corresponding encryption and masking configuration.
9. When configuring encryption, the shuffle data and configure the shuffling resources can be chosen immediately.
10. Click Batch Encryption Batch Masking Batch Encryption Masking to complete the batch encryption and masking. If selected immediate shuffle data, the data shuffling background task will be started. Click View Shuffle data Progress to view the shuffling progress. In the operation of the encrypted column in the table list, the ID of the shuffling and the progress of the shuffling also can be seen.

Masking Configuration #

Applicable Scenario

The user has sorted out the configurations that need to be masking. After adding the masking configuration, it can be used like a template.

Precautions

Console has started normally

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Table to enter the table list.
3. Click the Masking Configuration button in the upper right corner to enter the masking configuration page.
4. Edit the following content and click Confirm to complete the addition and editing of the masking configuration.

Encryption configuration #

Applicable Scenario

The user has sorted out the configurations that need to be encryption. After adding the encryption configuration, it can be used like a template.

Precautions

Console has started normally

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Table to enter the table list.
3. Click the Encryption Configuration button in the upper right corner to enter the masking configuration page.
4. Edit the following content and click Confirm to complete the addition and editing of the encryption configuration.

Delete Masking #

Applicable Scenario

The user needs to delete the masking configuration in the table

Precautions

The cluster can be accessed normally, there are existing logical database and tables, and the columns in the tables have been configured masking

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Table to enter the table list.
3. Select table
4. Click the Delete Masking button in the upper right corner to complete the deletion of the column configuration in the table

Sharding #

Add Sharding #

Applicable Scenario

Add sharding configuration to existing table.

Precautions

The cluster can be accessed normally. There is a logical database and a table. There is no data in the table and there is no sharding configuration in the table.

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the Add Shard button in the plug-in column to enter the page for adding shards.
4. Fill in the information, see the table below for specific parameters.

Sharding configuration: regular means formatting definition rules; custom means using DistSQL definition rules, and there is no need to define storage nodes in the regular way; choose between regular and custom configurations, and the corresponding configuration items will appear in whichever way you choose.

Customize

5. Click Confirm to complete adding the sharding plug-in configuration.

Modify shards #

Applicable Scenario

Add sharding plug-in configuration to existing table.

Precautions

The cluster can be accessed normally, there is a logical database, there are tables, there is no data in the tables, and there are sharding plug-in configurations in the tables.

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the Modify Sharding button in the plug-in column to enter the page for modifying sharding. Enter the page to modify the sharding plug-in, the configuration of the sharding plug-in can be seen. If it is a sharding plug-in added regularly, it will be displayed regularly; if it is a sharding plug-in added in a custom way, it will be displayed in a custom way.
4. Fill in the information, see the table below for specific parameters. Sharding plug-in configuration: regular means formatting definition rules; custom means using DIStSQL definition rules, and there is no need to define storage nodes in the regular way; choose between regular and custom configurations, and the corresponding configuration items will appear in whichever way you choose.

Customize

5. Click Confirm to complete the modification of the sharding plug-in configuration.

Encryption #

Add Encryption Configuration General Configuration #

Applicable Scenario

Add encryption plug-in configuration to the existing table.

Precautions

The cluster can be accessed normally. There are existing logical databases and tables. There are no encryption plug-in configurations in the tables, and no data shuffling is involved.

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the Configure Encryption button to enter the page for configuring encryption.
4. Click the New button to enter the page to create new encryption.
5. Fill in the information, see the table below for specific parameters.

Encryption plug-in configuration: Regular means formatting definition rules; Custom means using DIStSQL definition rules. Choose between regular and custom configurations. Whichever method is chosen, the corresponding configuration items will appear. [Use encrypted column query] Check, use the encrypted data column when querying; uncheck, use the data column before encryption when querying;

The default is checked;

When plaintext storage is unchecked, [Use encrypted column query] must be checked and cannot be changed;

When plain text storage is all checked, [Use encrypted column query] can be changed to unchecked.

Conventional

Customize

5. Click Confirm to complete adding the encryption plug-in configuration.

Configure encryption #

Applicable Scenario

Used for users to configure encryption for columns in tables

1. Enable and disable storage of plain text

Enable and disable the storage of plain text will increase or decrease fields in the database, so please operate with caution.

2. Enable and disable the use of plain text queries

Enable the use of plain text query: the plain text column query will be used

Disable using plain text query: cipher text column query will be used

3. Enable and disable fuzzy query

After enabling fuzzy query, the Like syntax can be used to query data on encrypted columns.

After disabling the fuzzy query, an error will be reported when querying data using Like syntax.

4. Modify encryption configuration

The encryption algorithm and key can be modified by modifying the encryption configuration. The ciphertext will be shuffled into plaintext first, and then the plaintext will be shuffled into ciphertext using the new key. Therefore, the prerequisite for using this function is to turn on storing plaintext and turn off using plaintext query.

5. Delete the encryption configuration of the current column

Precautions

No operations can be performed while the table is being configured for encryption. Click the yellow hourglass to see the progress of configuring encryption. Generally, the progress will be very fast. When using Oracle stored procedure encryption, the time to configure encryption will be relatively long.

Shuffle data #

Applicable Scenario

Check the shuffle progress

Precautions

No operations can be performed on the table being shuffled. The encryption configuration can be checked.

Operation Steps

1. Login to SphereEx-Console.
2. Click Cluster Management->Table to enter the table list.
3. Click Batch Encryption Masking and click Shuffle Data or directly click Shuffle Data Progress of the encrypted column of the table to query the shuffle progress. For tables with shuffle tasks, a yellow hourglass will appear next to the shuffle button.

View encryption configuration, delete encryption configuration, turn on and off plain text storage, turn on and off plain text query, turn on and off fuzzy query, modify encryption configuration, delete the encryption configuration of the current column

Masking #

Add masking configuration General configuration #

Masking configuration: Regular means formatting definition rules; Custom means using DistSQL to define rules, and there is no need to define storage nodes regularly; choose between regular and custom configurations, and the corresponding configuration items will appear in whichever way you choose.

Add masking configuration Custom configuration #

Modify the masking configuration of columns in the table #

1. Login to SphereEx-Console.
2. Click Cluster Management->Table to enter the table list.
3. Click the Modify button of the masking column in the table list to enter the masking modification page.。
4. Edit the following content and click Confirm to complete the modification of the masking configuration.

Delete the masking configuration in the table #

1. Login to SphereEx-Console.
2. Click Cluster Management->Table to enter the table list.
3. Click the Delete button of the masking column in the table list to complete the deletion of the masking configuration.

Table Group Management #

Applicable Scenario

Bind sharding plugins deployed with the same algorithm into a group for query optimization.

Precautions

The cluster can be accessed normally, there is an existing logical database, and the configuration of the table sharding plug-in is consistent

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the Table Group Management button in the upper right corner to enter the table group management page.
4. Fill in the information, see the table below for specific parameters.

5. Click OK to complete the table group management.

View table structure #

Applicable Scenario

Check the table structure in the form of create table, but you cannot see the plug-in configuration.

Precautions

The cluster can be accessed normally, and there are logic databases and tables.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the View Table Structure button in the operation column to enter the page for viewing the table structure.
4. Click Back to return to the list of table objects.

View Table Distribution #

Applicable Scenario

View the storage node where the table is located

Precautions

The cluster can be accessed normally, and there are logic databases and tables.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the View Table Distribution button in the operation column to enter the page for viewing table distribution.
4. Click Back to return to the list of table objects.

Modify Table #

Applicable Scenario

Modify table structure

Precautions

The cluster can be accessed normally, and there are logic databases and tables.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the Modify Table button in the operation column to enter the page for modifying the table.
4. Fill in the SQL for modifying the table structure, and you can click Example to view the statement example.
5. Click OK to complete the table modification.

Delete #

Applicable Scenario

Delete tables, delete plugin configurations, delete table structures and data.

Precautions

The cluster can be accessed normally, and there are existing logical databases and tables. The deletion is a physical deletion and cannot be recovered. Please operate with caution.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->Object to enter the table object list.
3. Click the Delete button in the operation column to enter the page for modifying the table.
4. Click OK to complete the table deletion.

User Privilege #

User

Create new user #

Applicable Scenario

Using DBPlusEngine’s Unified Privilege Management Function

Precautions

The cluster can be accessed normally, there is a logic database

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->User Privilege to enter the user list.
3. Click the Create New User button at upper right corner to enter the create new user page.
4. Fill in the information, see the table below for specific parameters.

Modify password #

Applicable Scenario

Change DBPlusEngine Dist user password

Precautions

The cluster can be accessed normally, there is a logic database.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->User Privilege to enter the user list.
3. Click the Change Password button in the action column Enter the new password and click Confirm to complete the password change

View Privilege #

Applicable Scenario

View DBPlusEngine Dist user privilege

Precautions

The cluster can be accessed normally, there is a logic database.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->User Privilege to enter the user list.
3. Click the View Privilege button in the action column to view user permissions

Manage Authorization #

Applicable Scenario

Adjust the privileges of DBPlusEngine Dist user

Precautions

The cluster can be accessed normally, there is a logic database.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->User Privilege to enter the user list.
3. Click the Manage Authorization button in the operation column to enter the authorization management page.

Delete #

Applicable Scenario

Delete DBPlusEngine Dist user.

Precautions

The cluster can be accessed normally, and there are existing logic databases and tables.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Cluster Management->User Privilege to enter the user list.
3. Click the Delete button in the operation column to complete the deletion of the user

Role

Create new role #

Applicable Scenario

Use the unified privilege management function of DBPlusEngine to manage user privilege by roles

Precautions

The cluster can be accessed normally, there is a logic database.

Operation Steps

1. Log in to SphereEx-Console.
2. Log in to SphereEx-Console. Click Cluster Management->User Privilege->Role to enter the role list.
3. Click the New Role button in the upper right corner to enter the page for creating a new role.
4. Fill in the information, see the table below for specific parameters.

View Privilege #

Applicable Scenario

View Role’s Privilege

Precautions

The cluster can be accessed normally, there is a logic database.

Operation Steps

1. Log in to SphereEx-Console.
2. Log in to SphereEx-Console. Click Cluster Management->User Privilege->Role to enter the role list.
3. Click the View Privilege button in the action column to view the role privilege

Manage Authorization #

Applicable Scenario

Adjust role privilege

Precautions

The cluster can be accessed normally, there is a logic database.

Operation Steps

1. Log in to SphereEx-Console.
2. Log in to SphereEx-Console. Click Cluster Management->User Privilege->Role to enter the role list.
3. Click the Manage Authorization button in the operation column to enter the authorization management page.

Delete #

Applicable Scenario

Delete Role

Precautions

The cluster can be accessed normally, there is a logic database.

Operation Steps

1. Log in to SphereEx-Console.
2. Log in to SphereEx-Console. Click Cluster Management->User Privilege->Role to enter the role list.
3. Click the Delete button in the operation column to complete the role deletion

SQL Management #

Slow Log #

Applicable Scenario

Check the slow logs generated by the cluster

Precautions

• The cluster can be accessed normally
• slow log is turned on
• Slow logs can be filtered by SQL statement, DistSQL user, computing node, fuzzy query; time, logic database, SQL type, execution time, and equivalent query.

Operation Steps

1. Log in to SphereEx-Console.
2. Click SQL Management->Slow Log to enter the Slow Log page. The content of the slow log includes: time, user, computing node, logic database, SQL type, execution time, and SQL statement.

Hover to View SQL Routing to jump to the SQL Routing page to view slow SQL.

3. Click Export to export all logs that meet the conditions

SQL Routing #

Applicable Scenario

View the execution path of SQL

Precautions

The cluster can be accessed normally

Operation Steps

1. Log in to SphereEx-Console.
2. Click SQL Management->SQL Routing to enter the SQL Routing page.
3. Select the cluster, select the logic database, enter the SQL statement, and click the View SQL Routing button to view the execution routing.

SQL Audit #

Applicable Scenario

Audit after the event by viewing all SQL executed by the cluster.

Precautions

• The cluster can be accessed normally
• audit logging is turned on
• Audit logs can be filtered by SQL statement, DistSQL user, computing node, fuzzy query; time range, SQL type, equivalent query.

Operation Steps

1. Log in to SphereEx-Console.
2. Click SQL Management->SQL Audit to enter the SQL Audit Log page. The content of the audit log includes: time, DistSQL user, computing node, logic database, SQL type, and SQL statement.
3. Click Export to export all logs that meet the criteria.

Log Management #

View logs through SphereEx-Console #

Applicable Scenario

By viewing the running logs of the cluster computing nodes.

Precautions

• The cluster can be accessed normally
• You can use keyword fuzzy query; time range equivalent query to filter running logs.

Operation Steps

1. Log in to SphereEx-Console.
2. Click Log Management to enter the log management page.
3. Click Export to export all logs that meet the criteria.

Log Management Related Parameters #

system-log-level System log level, non-built-in loggers use this configuration system-log-pattern system log format, global common

DistSQL variable modification example

SET DIST VARIABLE system_log_level = DEBUG
SET DIST VARIABLE system_log_pattern = '[%-8level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %logger{39} - %msg%n'

To adjust the running log parameters through SphereEx-Console, please refer to [Cluster parameter management](/docs/operation-maintenance/2-non-containerized/cluster-management/#Parameter Management)

Adjust the running log through DistSQL, please refer to[Log management](/docs/operation-maintenance/7-log-management/#Cluster Log)

For parameter details, please refer to [Cluster Parameter List](/docs/reference/parameter/#Cluster Parameter List)

Sensitive data identification #

Introduction #

The figure below shows the entire data life cycle and sensitive data identification. SphereEx-SDD （SDD for short）It solves the first phase of the problem, helping users quickly identify sensitive data risks, giving security specifications and feasible implementation suggestions, and accelerating the implementation of user security transformation work.

Install under Linux x86 enviroment #

download: curl -O -u download_1fNAVPTXxm:gyKxRvFkBEKkTYCT https://artifacts.sphere-ex.cn/tar/repository/download/sphereex_sdd/cn/sphere-ex/xxx/1.x.x/sphereex-sdd-1.x.x.tar.gz

Please contact sales for download link 400-900-2818

Unzip the package

Enter the directory: cd sdd

Start： ./start.sh

The default port is 3000, which can be modified by editing start.sh

sdd oracle client dependency library installation：

Method 1: Log in to Oracle’s official website to download client dependencies and use rpm to install them offline.

URL: https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html

rpm -ivh oracle-instantclient-sqlplus-21.10.0.0.0-1.x86_64.rpm
rpm -ivh oracle-instantclient-basic-21.10.0.0.0-1.el8.x86_64.rpm
rpm -ivh oracle-instantclient-devel-21.10.0.0.0-1.x86_64.rpm

Method 2: Install online using yum

yum install oracle-instantclient-sqlplus
yum install oracle-instantclient-devel
yum install oracle-instantclient-basic

Docker Installation #

docker run -it -d -v /xxx:/work/data -p3000:3000 artifacts.sphere-ex.com/xxx/sphereex-sdd:.x.x.x web

Please contact sales for download link 400-900-2818

Using #

Login #

Tool URL: http://xxx.xxx.xxx.xxx:3000/ Built-in user：admin Built-in password：admin

Rule management #

Built-in Rule

There are currently 30 built-in rules, covering most personal information and some sensitive information.

Customized rules

Users can customize the rules according to their own data scenarios.

Operation Steps：

1. Click Add Rule
2. Fill in the rule name, select the data classification, support field name list and field name regular expression to configure field mapping, and fill in the rule description.
3. Optional description according to the situation
4. Click OK

Built-in rule group

Currently, two standard rule groups, GDPR and GB/T 35273, are built-in, each containing built-in rules for users to use directly.

Custom rule group

Users can create a customized rule group and select one or more rules for targeted identification.

Operation Steps:

1. Click Add Rule Group
2. Fill in the rule group name and select the rule
3. Click Confirm

Create a copy of built-in rules to facilitate users to add or delete rules in existing rule groups

Operation Steps：

1. Click Create Copy in the built-in rule group operation
2. Fill in the rule group name and adjust the rules
3. Click Confirm

Set up rules in a rule group

The built-in field mapping, custom field mapping, data classification, and security recommendations of rules can be configured in the rule group.

Data Source Management #

Add corresponding data sources for data identification. Currently, it supports MySQL, Oracle, SQLServer, PostgreSQL, Presto, and Hive. After adding a data source, click “New Identification Task” directly in the operation column to create a task directly.

Operation Steps

1. Click Add
2. Fill in the information
3. Click Confirm

System Configuration #

1. License Configuration

Paste the License-Key authorized by SphereEx into the text field and click Confirm to complete the license registration.

2. Backup and restore

Custom rules, rule groups, and data sources can be chosen to back up. Suitable for daily backup and system upgrades

Task Management #

Configure execution recognition tasks

Operation Steps：

1. Click Add
2. Select data source and rule group
3. Select execution mode
4. Fill in the remark information as needed
5. Choose whether to execute immediately
6. Click OK

• Task mode description

Extremely fast mode: fetch 50,000 pieces of data from each table, 5 concurrent executions

Full mode: fetch full data from each table, fetch 5W pieces of data each time, and execute serially

Custom mode: user-defined concurrency and sampling

• Custom mode parallelism configuration

• Custom mode parallelism configuration sampling configuration

Sampling for configuring identification tools

Random sampling: Specify the sampling ratio and randomly sample according to the specified sampling ratio.

Specify the number of sampling items: Specify the number of rows to scan, and take the specified number of items according to the paging algorithm of the database.

Full sampling: full scan

After the task execution is completed, an identified report will be generated, which can be viewed on the report management page. Executing a task multiple times will generate multiple reports.

Report management #

Click View Report in the operation column to view the corresponding report.

In the report, users can adjust the four configurations of matching threshold, interpretation standard, and recognition result display to obtain the corresponding report results.

• Matching degree: After the recognition task is completed, the rule matching degree of each field is obtained.
• Match threshold: User-defined recognition results that can be displayed in the report
• After modifying the above configuration, click “View Report” to view the report.
• Full-text fuzzy search is possible, and field names, matching degrees, recognition results, data classification, and security recommendations can be sorted.
• Click “Download Report” to download the report locally. Currently, two report types, excel and html, are supported.

Connect Cluster #

Users can log in to the cluster in several ways:

• SphereEx-Console: Users can use SphereEx-Console to complete management actions through the GUI interface after login (built-in restricted DistSQL, SQL terminal is not yet available).
• Database Command: For example, MySQL Client, PG PSQL, log in to any computing node in the cluster. Use USE to locate the logical database and execute SQL. Example usage: mysql -uroot -p -P3307 -h127.0.0.1
• Third Database Tools: Tools such as navicate, some functions are limited, Todo to be tested.