Distributed View in WSO2 Status Dashboard

As a second project of my internship at WSO2, I got an opportunity to work on the status dashboard for WSO2 Stream Processor.

What is WSO2 Status Dashboard?

The Status Dashboard of WSO2 Stream Processor allows user to monitor the metrics of a stand-alone WSO2 Stream Processor instance or a WSO2 Stream Processor cluster. This involves monitoring whether all processes of the setup are working in a healthy manner, monitoring the current status of a Stream Processor node, and viewing metrics relating to the history of a node or the cluster. Both JVM level metrics or Siddhi application level metrics can be viewed from the monitoring dashboard.

We can understand the performance of the system via

• Throughput
• Latency
• CPU, memory utilizations

We can Monitor in various scales

• Node Level
• Siddhi App Level
• Siddhi Query Level

Okay, hope you all got a basic idea of the WSO2 Status dashboard and its functionalities. Monitoring the Distributed setup using WSO2 Status Dashboard is a new feature of status dashboard.

So What is Distributed Set Up?

Let me explain a basic idea of distributed setup.

The most common deployment pattern for WSO2 Stream Processor is the HA Deployment (Minimum High Availability Deployment) that offers high availability and high performance with the minimum amount of resources. However, there are a some real time scenarios where the HA (High Availability) deployment is not sufficient to handle the scenarios. The Fully Distributed Deployment pattern is introduced to handle to such scenarios. The main benefits to this approach are high availability and fault-tolerance.

In simple words, a distributed system is that splits into multiple running instances, all instances work together in a cluster to appear as a single instance to the end user.

Distributed deployment pattern is suitable in such scenarios where single SP instance fails to manage high amount of data as it allows complex event processing in a distributed manner. eg: High volume of data will be distributed among multiple SP instances instead of having them accumulated at a single instance. For detailed information about distributed system, refer here.

Monitoring Distributed System in WSO2 Status Dashboard

Hope you all got an idea about distributed system and wso2 status dashboard. Let’s move on how we can use status dashboard to monitor distributed system.

Before we start:

• Fully distributed system must be available.
  To configure the distributed system, Please follow the instructions mentioned here
• Download the latest WSO2 Stream Processor
  You can find the latest pack here

How to start WSO2 Status Dashboard?

1. Extract the downloaded zip and navigate to the <SP_HOME>/bin directory (<SP_HOME> is the extracted directory).
2. Issue the following commands to start the WSO2 Status Dashboard.

• For Windows: dashboard.bat
• For Linux: ./dashboard.sh

Once WSO2 dashboard is successfully started, a log similar to the following is printed in the CLI.

The server log prints the Status Dashboard URL in the start up logs as shown below.

Let’s get started!

• Access the WSO2 Status Dashboard via the https://<HOST_NAME>:<STATUS_DASHBOARD_PORT>monitoring status dashboard URL.
  The default URL is: https://192.168.1.2:9643/monitoring
• The Stream Processor Studio opens as shown below.

• You can login to status dashboard by giving ‘admin’ as a user name and password
• Once you login to the status dashboard, the nodes that are already added to the Status Dashboard are displayed
  If no nodes are displayed, you can add nodes for which you want to view the status by following the procedure below:

• Click ‘Add New Node’ button, this opens the following dialog box.

• Enter the following information in the dialog box and click ADD NODE to add a gadget for the required node in the Node Overview page.

1. In the Host parameter, enter the HTTPS host ID of the node you want to add.
2. In the Port parameter, enter the HTTPS port number of the node you want to add.

If the node you added is reachable, automatically status dashboard determine the runtime environment and will proceed further or else the following dialog box will be opened.

Since this blog is all about the Distributed view, am gonna add manager node :)

Node View

The following basic details are displayed for each node.

• CPU Usage: The CPU resources consumed by the SP node out of the available CPU resources in the machine in which it is deployed is expressed as a percentage.
• Memory Usage: The memory consumed by the node as a percentage of the total memory available in the system.
• Load Average: The average system load calculated over a given period of time of 1, 5 and 15 minutes
• Siddhi Apps: The total number of Siddhi applications deployed in the node.
  * Green color indicates active number of siddhi applicatios
  * Red color indicates inactive number of siddhi applications

This dispalys the change that has taken taken place in the CPU usage, memory usage and the load average of nodes since the status was last viewed in the status dashboard.

Positive changes are indicated in green (e.g., a decrease in the CPU usage in the above example), and Negative changes are indicated in red (an increase in the memory usage and the load average in the above example).

Viewing status details

Sample View

Description

The nodes that are connected in the distributed deployment are displayed under the relevant group ID in the status dashboard (e.g : ‘sp’ in the above example).

Managers
The active manager node in the cluster is indicated by a green dot that is displayed with the host name and the port of the node. Similarly, a grey dot is displayed for passive manager nodes in the cluster.

Workers
When user adds the active manager node, automatically it retrieves the worker node details that are connected with that particular deployment and you can see the metrics of each worker nodes

Purpose

This allows you to determine the following:

• Whether the request load is efficiently allocated between the nodes of a cluster.
• Whether the cluster has sufficient resources to handle the load of requests.
• What are the nodes connected with the particular deployment

Recommended Action

• If there is a disparity in the CPU usage and the memory consumption of the nodes, redeploy the Siddhi applications between the nodes to balance out the workload.
• If the CPU and memory are fully used and and the request load is increasing, allocate more resources (e.g., more memory, more nodes, etc.).

Viewing Parent siddhi application details

When you open the WSO2 SP Status Dashboard, the Node View page is displayed by default. To view information specific to a selected acive manager node, click on the relevant gadget. Passive manager node gadgets are disabled.

This opens a separate page for the manager node as shown in the example below

Description

The Active/Deployed parent siddhi application deployed in the cluster is indicated by a green dot that is displayed with the name of the parent siddhi application. Similarly, a orange dot is displayed for Inactive/ Un-deployed siddhi applications in the cluster.

• Worker Nodes : This gives the details of worker nodes that are inside resource cluster.
  Yellow Color indicates total number of worker nodes in the resource cluster (Eg: 2 in the above example which means for that particular distributed deployment we have two worker nodes in the resource cluster).
  Green Color indicates the number of worker nodes used in the particular parent siddhi application deployment. (Eg : 1 in the above example which means even though we have two worker nodes in the resource cluster,only one work node used for this specific parent siddhi application’s deployment).
• Child Apps : This gives the total number of child apps of the particular parent siddhi application
  Green Color indicates number of deployed child apps
  Red Color indicates number of un-deployed child apps
• Groups : This gives the total number of execution groups of that parent siddhi application. (Eg : In the above example, Testing Parent siddhi application contains 1 execution groups)

Purpose

This allows you to determine the following:

• Summary details of each parent siddhi applications that are inside the specific active manager node.

Viewing Specific Parent Siddhi Application

To view information specific to a Parent Siddhi application deployed in the active manager node, click on the relevant parent Siddhi application gadget.

This opens a page with information specific to the selected Parent Siddhi application as shown in the example below.

Code View

View

Description

This displays the queries defined in the Parent Siddhi file of the application.

This allows you to check the queries of the Siddhi application if any further investigations are needed based on the kafka diagrams and performance.
For detailed instructions to write a Siddhi application, see Converting to a Distributed Streaming Application.

For detailed information about the Siddhi logic, see the Siddhi Query Guide.

Distributed Siddhi App Deployment

View

Description

This displays how kafka topics connected with each child siddhi applications.
Red color indicator box indicates the kafka topic name.
Blue color indicator box indicates the child siddhi applications

Let me explain above example,
Kafka topic ‘topic_1’ is a source, that publishes events to ‘SampleTest-001–1’ and ‘SampleTest-001–2’ child apps. Then both child apps publishes events to ‘SampleTest.filteredStockStream’ kafka topic. Then ‘SampleTest.filteredStockStream’ publishes events to SampleTest-004–1child app. Finally SampleTest-004–1 child app publishes events to kafka topic ‘topic_2’.

Child App Details

View

Description

This table displays the complete list of Child Siddhi applications in the selected parent siddhi application. The status is displayed in green for active Siddhi applications, and in red for inactive Siddhi applications. In addition, the following is displayed for each Siddhi application:

• Group Name : The name of the group that particular child siddhi application belongs to
• Worker Node: HTTPS host and HTTPS port of the worker node where that particular child siddhi application is deployed. By clicking that host and the port, you can get the worker specific details. This blog mainly focus on the manager view. To monitor each worker nodes that are connected with the distributed setup, please refer here

Okay now, It’s time for live demo :)

That’s all :)

Hope you all understand how we can use WSO2 Status Dashboard to monitor Distributed Setup :)