Monitor and Alarm #

Description:

1. Monitoring can only be completed through SphereEx-Console.
2. Currently, there is no alarm function.

Cluster ecosystem tool monitoring #

Currently, SphereEx-Console and SphereEx-Boot have no monitor function.

Monitoring of cluster components #

Host monitoring #

• Monitoring indicators

• View Monitoring

  Applicable Scenarios

  View host monitoring.

  Notes

  • The monitoring center is functioning normally.
  • The database has been configured for monitoring.

  Steps

  1. Log in to SphereEx-Console.
  2. Click Monitoring -> Hosts to enter the monitoring list.
  3. Click the Monitoring button in the action column to view the monitoring indicators.

• Configure Monitoring

  Applicable Scenarios

  Configure host monitoring.

  Notes

  • The monitoring center is functioning normally.
  • The Governance Center host has been registered and has read-write permissions for the monitoring plugin installation directory.

  Steps

  1. Log in to SphereEx-Console.
  2. Click Resources -> Hosts to enter the host list.
  3. Click Configure Monitoring in the operation column to enter the monitoring configuration page.
  4. Configure the monitoring information as follows.

5. Click Install to complete the installation of the monitoring plugin and the configuration of the monitoring center.

Database Monitoring #

• Monitoring Metrics

• Viewing Monitoring

  - Applicable Scenarios

  View database instance monitoring

  Notes

  • The monitoring center is functioning normally.
  • The database has been configured for monitoring.

  Operation Steps

  1. Log in to SphereEx-Console.
  2. Click Monitoring->Database to enter the monitoring list.
  3. Click the Monitoring button in the operation column to view the monitoring indicators.

• Configuring Monitoring

  Applicable Scenarios

  Configure database monitoring

  Notes

  • The monitoring center is functioning normally.
  • The Governance Center host has been registered and has read-write permissions for the monitoring plugin installation directory.

  Operation Steps

  1. Log in to SphereEx-Console.
  2. Click Resource->Database to enter the host list.
  3. Click Configure Monitoring in the operation column to enter the monitoring configuration page.
  4. Configure the monitoring information as follows.

5. Click Install to complete the installation of the monitoring plugin and configuration of the monitoring center.

Governance Center Monitoring #

• Monitoring Metrics

• Viewing Monitoring Metrics

  Applicable Scenarios

  View monitoring metrics for Governance Center instances.

  Note

  • The monitoring center is functioning normally.
  • Governance Center monitoring is configured.

  Procedure

  1. Log in to SphereEx-Console.
  2. Click Monitoring -> Governance Center to enter the Governance Center list.
  3. Click Monitoring Node in the Operations column to enter the monitoring list.
  4. Click the Monitor button in the Operations column to view the monitoring metrics.

• Configuring Monitoring

  Applicable Scenarios

  Configure monitoring for Governance Center.

  Prerequisites

  • The monitoring center is functioning normally.
  • The Governance Center host has been registered and has read-write permissions for the monitoring plugin installation directory.

  Procedure

  1. Log in to SphereEx-Console.
  2. Click Resources > Governance Center to enter the host list.
  3. Click Configure Monitoring in the Operations column to enter the monitoring configuration page.
  4. Configure the monitoring information as follows.

5. Click Install to complete the installation of the monitoring plugin and the configuration of the monitoring center.

Monitoring Center Monitoring #

• Monitoring Metrics

• View Monitoring

  Applicable Scenarios

  View monitoring data of the monitoring center itself

  Note

  Monitoring center is running properly

  Steps

  1. Log in to SphereEx-Console.
  2. Click Monitoring > Monitoring Center to enter the monitoring list.
  3. Click the Monitoring button in the operation column to view the monitoring indicators.

• Configure Monitoring

  The monitoring of the monitoring center is enabled by default when it is installed, and no additional configuration is required.

Log Center Monitoring #

Todo

This feature will be implemented in Console version 1.2.

Cluster Monitoring #

• Monitoring indicators

• View monitoring

  Applicable scenarios

  View cluster monitoring.

  Notes

  • Monitoring center is functioning normally.
  • Governance Center has already been configured for monitoring.

  Operation steps:

  1. Log in to SphereEx-Console.
  2. Click Monitoring->Cluster to enter the monitoring list.
  3. Click the Monitoring button in the operation column to view the monitoring indicators.

• Configure monitoring

  Applicable scenarios

  Configure cluster monitoring, which actually configures the monitoring of the computing nodes.

  Notes

  • Monitoring center is functioning normally.
  • The host for the Governance Center has been registered and has read and write permissions for the monitoring plugin installation directory.

  - Operation steps

  1. Log in to SphereEx-Console.
  2. Click Cluster Management->Cluster to enter the host list.
  3. Click Configure Monitoring in the operation column to enter the monitoring configuration page.
  4. Configure the monitoring information as follows.

5. Click on Add Monitoring Configuration to complete the monitoring center configuration.

Compute node monitoring #

• Monitoring indicators

• View monitoring

  Application Scenario

  View cluster monitoring.

  Notes

  • Monitoring center is functioning normally.
  • Governance Center has already been configured for monitoring.

  Steps

  1. Log in to SphereEx-Console.
  2. Click Monitoring -> Cluster to enter the cluster list.
  3. Click Node Monitoring in the operation column to enter the compute node list.
  4. Click Monitoring in the operation column to view monitoring indicators.

• Configure monitoring

  Complete by configuring cluster monitoring.。

Storage node monitoring #

Currently, monitoring of storage nodes is not supported.

Agent management #

Introduction to Agent #

Background

In order to grasp the distributed system status, observe running state of the cluster is a new challenge. The point-to-point operation mode of logging in to a specific server cannot suite to large number of distributed servers. Telemetry through observable data is the recommended operation and maintenance mode for them. Tracking, metrics and logging are important ways to obtain observable data of system status. APM (application performance monitoring) is to monitor and diagnose the performance of the system by collecting, storing and analyzing the observable data of the system. Its main functions include performance index monitoring, call stack analysis, service topology, etc. DBPlusEngine is not responsible for gathering, storing and demonstrating APM data, but provides the necessary information for the APM. In other words, DBPlusEngine is only responsible for generating valuable data and submitting it to relevant systems through standard protocols or plug-ins. Tracing is to obtain the tracking information of SQL parsing and SQL execution. DBPlusEngine provides support for SkyWalking，Zipkin，Jaeger and OpenTelemetry by default. It also supports users to develop customized components through plug-in.MMMMMM

• Use OpenTelemetry

OpenTelemetry was merged by OpenTracing and OpenCencus in 2019. In this way, you only need to fill in the appropriate configuration in the agent configuration file according to OpenTelemetry SDK Autoconfigure Guide，Data can be exported to Jaeger, Zipkin.

• Use SkyWalking

Cooperating with Apache SkyWalking team, DBPlusEngine team has realized ShardingSphere automatic monitor probe to automatically send performance data to SkyWalking. Note that automatic probe in this way cannot be used together with DBPlusEngine plug-in probe.

Metrics used to collect and display statistical indicator of cluster. DBPlusEngine supports Prometheus by default.

Challenges

Tracing and metrics need to collect system information through event tracking. Lots of events tracking make kernel code mess, difficult to maintain, and difficult to customize extend.

Goal

The goal of Apache ShardingSphere observability module is providing as many performance and statistical indicators as possible and isolating kernel code and embedded code.

Core concepts

• Agent

Based on bytecode enhancement and plugin design to provide tracing, metrics and logging features.

• APM

APM is an acronym for Application Performance Monitoring. Focusing on the performance diagnosis of distributed systems, its main functions include call chain display, application topology analysis, etc.

• Tracing

Tracing data between distributed services or internal processes will be collected by agent. It will then be sent to third-party APM systems.

• Metrics

System statistical indicators are collected through probes for writing in timing database and display by third-party applications.

Agent Configuration #

Agent Configuration

• Directory Structure

Create an agent directory and extract the agent-bin package into the agent directory.

mkdir agent
tar -zxvf apache-shardingsphere-${latest.release.version}-shardingsphere-agent-bin.tar.gz -C agent
cd agent
tree 
.
├── LICENSE
├── NOTICE
├── README.txt
├── conf
│   └── agent.yaml
├── plugins
│   ├── lib
│   │   ├── shardingsphere-agent-metrics-core-${latest.release.version}-SNAPSHOT.jar
│   │   └── shardingsphere-agent-plugin-core-${latest.release.version}-SNAPSHOT.jar
│   ├── logging
│   │   └── shardingsphere-agent-logging-file-${latest.release.version}-SNAPSHOT.jar
│   ├── metrics
│   │   └── shardingsphere-agent-metrics-prometheus-${latest.release.version}-SNAPSHOT.jar
│   └── tracing
│       ├── shardingsphere-agent-tracing-opentelemetry-${latest.release.version}-SNAPSHOT.jar
│       └── shardingsphere-agent-tracing-opentracing-${latest.release.version}-SNAPSHOT.jar
├── shardingsphere-agent-${latest.release.version}-SNAPSHOT.jar
└── template
    ├── dbplusengine-driver-grafana-template.json
    └── dbplusengine-proxy-grafana-template.json

• Configuration

The file agent.yaml is the configuration file, and the available plugins include Jaeger, OpenTracing, Zipkin, OpenTelemetry, Logging, and Prometheus. To enable a plugin, simply comment out the corresponding plugin name in ignoredPluginNames.

plugins:
  logging:
    File:
      props:
        slow-query-log: true
        long-query-time: 5000
        general-query-log: true
  # metrics:
  #   Prometheus:
  #     host: "localhost"
  #     port: 9090
  #     props:
  #       jvm-information-collector-enabled: "true"
  # tracing:
  #   OpenTelemetry:
  #     props:
  #       otel.service.name: "shardingsphere"
  #       otel.traces.exporter: "jaeger"
  #       otel.exporter.otlp.traces.endpoint: "http://localhost:14250"
  #       otel.traces.sampler: "always_on"

• Parameter Description

More info aboutOpenTelemetry refers OpenTelemetry SDK Autoconfigure Guide

Using Agent in DBPlusEngine-Proxy #

Start Agent at the same time as Start Proxy

bin/start.sh -g

Normal startup logs can be viewed in the corresponding DBPlusEngine-Proxy logs, and Metric and Tracing data can be viewed through the configured address after accessing the Proxy.

Agent Metrics

DBPlusEngine-Driver 中使用 Agent #

Start Agent

• Prepare projects that integrate DBPlusEngine-Driver, such as SpringBoot projects

• Add javaagent configuration at startup

java -javaagent:/xxx/shardingsphere-agent-${latest.release.version}.jar -jar spring-boot-dbplusengine-driver-test.jar

Agent Metrics