VM status monitoring

From ISPWiki
Jump to: navigation, search
Hierarchy: VMmanager KVM -> VIRTUAL MACHINES
VMmanager Cloud -> VIRTUAL MACHINES

Purpose

The VM status monitoring subsystem is specially designed for sending information about virtual machines from libvirt to VMmanager KVM/Cloud. The main goals of this subsystem:

  • Synchronizes data in the control panel about virtual machines and their actual statuses
  • Transmits statuses of virtual machines with minimal time lags
  • Registers changes of VM statuses

Monitoring subsystem:

  • Participates in OS installation on a virtual machine
  • Controls duplicated of virtual machines

Subsystem architecture

Subsystem consists of 2 services: vmwatch-master и vmwatch-node.

A message queue, which is implemented using ZeroMQ library via the PUSH-PULL pattern, is used for interaction between services.

vmwatch-master

This service runs on the node, where the control panel is installed

It receives messages on the IP address and port specified in the configuration file, processes them, and calls the corresponding API functions of a control panel functions to send data about changes in statuses of virtual machines.


VMmanager Cloud: when cloud functions are turned on, if the master server changes, the service is stopped and blocked on the old master server. All the services that run vmwatch-master are configured on the new server, and the system reconfigures all vmwatch-master services in order to send message to the new master server.

vmwatch-node

The service runs on every cluster node. It sends messages to the IP address and port specified in the configuration file. It receives information about changes of VM statuses from libvirtd. If connection with libvirt is lost, the service will be stopped.

Service configuration

System V (System D for CentOS 7) controls service run and autorun. Script setting is executed during VMmanager package installation or upgrade to 5.86 version

vmwatch-master service and its control script are kept in vmmanager-kvm package for VMmanager KVM and in vmmanager-cloud package for VMmanager Cloud. vmwatch-node service and its control script are kept in vmmanager-pkg-vmnode package

Services run automatically during OS loading or configuration process.

After VMmanager installation vmwatch-master service is blocked until the first cluster node is added into the cluster.

Service work parameters are recorded in vmwatch-master.conf and vmwatch-node.conf configuration files, which are kept in /usr/local/mgr5/etc catalogue. Configuration files are created by the control panel automatically.

The configuration file is recorded in the NAME_PARAMETER=VALUE format, one parameter on one line.

vmwatch-master.conf configuration file

It may contain the following parameters:

  • disabled - If parameter has yes value, it blocks service execution.
  • listen - Parameter value in 'protocol://IP:port' format assigns socket where service recieves messages.
  • secret - Undefined symbol line (A-Z, 0-9), parameter value is used for incoming messages authentification.

vmwatch-node.conf configuration file

It may contain the following parameters:

  • connect - Parameter value in 'protocol://IP:port' format assigns socket where service sends messages
  • secret - Parameter value is used for incoming messages authentification. It has to match with secret parameter value of vmwatch-master configuration file
  • node_id - Cluster node identificator which is used in messages for message source identification

Service configuration

Service setting is executed by the control panel in automatic mode if it's necessary:

  • During the upgrade to the version which supports monitoring subsystem, and if there are nodes in the cluster.
  • When a cluster node is added into the cluster
  • During master node migration to the another cluster node (only for VMmanager Cloud if cloud functions are enabled)

IP address assignment, on which vmwatch-master receives messages, is executed in the following way:

  • VmwatchListenIp parameter value of configuration file is being checked. Only for VMmanager KVM
  • If IP address is not assigned by the parameter, a cluster node is searched, where the control panel is installed.
  • If there is no local cluster node, IP address scanning is started at all network interfaces of the server where the control panel is installed, it chooses the first IP address from the list which is not a local one.

Port where vmwatch-master receives messages, is defined by the VmwatchListenPort parameter of the control panel configuration file. Value by default is 1515.

To change IP address and/or port write the relevant parameters in the control panel configuration file, restart the panel, and call API vmwatch.configure function. This function configures all monitoring services according to the indicated parameters.

Service status control

vmwatch-master service status control is run by the panel when periodic API function is called. It is run by the cron each 60 minutes.

If control function detects that service is stopped, it will register the notification about this problem when this problem is being solved, there will be done a repeated configuration of the whole monitoring subsystem

vmwatch-node service status control is run by the panel when periodic API function is called. It is run by the cron each 5 minutes during the process of cluster node control panel status check (vmmini).

If the service is stopped, control function registers the notification about this problem when this problem is being solved, there will be done a repeated configuration on this cluster node

VM status check

In the "Cluster settings/Policy" module you can enable the check of VM statuses.

The system will use statistics data sent by VM monitoring services. The status check function analyses packages that were sent over the network and the number of write requests to a VM disk. Every 10 minutes vmwatch-node collects statistics from all domains and sends it to vmwatch-master. If any of the values equals 0, the corresponding warning will be displayed in the list of virtual machines.

Service log

vmwatch-master service log is kept in the /usr/local/mgr5/var/vmwatch-master.log file, vmwatch-node service log is kept in the /usr/local/mgr5/var/vmwatch-node.log file.