How to connect storages

From ISPWiki
Jump to: navigation, search
Hierarchy: VMmanager KVM -> Cluster configuration
VMmanager Cloud -> Cluster configuration

Storage of virtual disk images

Every virtual machine uses a virtual disk which is an image of the hard drive connected to it. Virtual disks are stored in a local or network storage. One virtual machine can have one or a few virtual disks connected and located in the same or different storages.

Storage can be added to cluster nodes from the section "Cluster settings" -> "Storage templates". Per default, it uses File system template with the name "File". When adding a new storage, you need to specify its type, access to it, and settings unique for this type of storage. VMmanager is homogenous; it means that all cluster nodes are to have the same sets of storages. When a new storage is added, the control panel adds it to all cluster nodes automaically if the network storage is used, or creates a storage on all cluster nodes if the local storage is utilized.

«Connect a new storage»

VMmanager supports the following storage types:

  • Name — enter the storage name;
  • Type — select the storage type:
    • File system;
    • LVM;
    • Network LVM;
    • iSCSI;
    • RBD;
    • GlusterFS;
    • NFS.
  • Directory on the cluster node — the path to the directory where virtual disk images will be stored;
  • Disk format — the format of the virtual disks that will be stored in the storage;
    • qcow2;
    • raw.
  • Reserved — a part of the storage reserved for system needs;
  • Units of measure — units of measure for the reserved space:
    • Percent;
    • MiB;
    • GiB.
  • Caching type — select a caching type which is used for virtual machines which disks are located in the storage:
    • defaul — writethrough;
    • none — caching disabled;
    • writethrough — data is written to the cache and the backing store location at the same time;
    • writeback — data is written to the cache and then I/O completion is confirmed. The data is then typically also written to the backing store in the background but the completion confirmation is not blocked on that.;
    • directsync — the child system uses writethrough, the host doesn't use caching. Every write operation performs fsync. Make sure the QEMU version supports this caching type;
    • unsafe — the data is cached by the hypervisor and data synchronization requests from the quest machine are ignored. There is a high provability to lose your data.
  • A|O mode — asynchronous input-output mode:
    • default — "native". You need to disable caching on the host-server (the caching type is "none" or "directsync");
    • threads — we recommend using it for file storages.
  • Description — additional information. It is displayed in the list of storage templates --> the "Status" column.

We recommend creating your main virtual disk storing your operating system and not crucial data in the local storage. The critical data is better to be stored in the network storage. Network storages are always remote. It allows to save space on your cluster nodes, provides better fault tolerance, and simplifis migration of virtual machines. However, normally the network storage is not as fast as the local one.

Storage format

The control panel supports the following formats for virtual disks of virtual machines:

  • RAW is a data format with unprocessed data or data with minimum processing. With this format, a virtual machine takes as much space as it was given during its creation;
  • Qcow2 is a disk image format of QEMU. A virtual machine takes as much space as it has data on it.

We recommend using Qcow2.

It is possible to convert storage from RAW to Qcow2. You can do it by following the article VMmanager:Converting storage into qcow2.

Local storage

File system (DIR)

Normally, this storage has one section in one disk.


Logical volume manager is a subsystem allowing to use different areas of the same hard drive and/or areas of different hard drives as one logical volume.

Size of file systems from different logical volumes is not limited with just one disk, as the volume can be located in different disks and sections.

LVM can have the following designations:

  • PV (Physical Volume) is a physical volume (disk sections or unsectioned disks);
  • VG (Volume Group) is a group of volumes (a few physical volumes (PV) united under the same group and building a single disk);
  • LV (Logical Volume) is a section created in physical space of a group of volumes (VG).

When a new LVM storage is created, the control panel checks whether there is a group of volumes (VG) with the specified storage name. If there is such group, then storage is initiated. If there is no such group, VMmanager will search for sections and hard drives allowing to create a group of volumes. A group of volumes can be located in the following sections and hard drives:

  • hard drive without any section;
  • hard drive section with linux-lvm inside.

If such sections or hard drives are not found, VMmanager will not be able to create an LVM storage on the physical server.

LVM storage can only support RAW format for virtual disks.

Read more in LVM documentation

Comparison of DIR and LVM:

Volume size Is given as a whole Can be increased once getting filled Can be increased or decreased
Management Easy Easy Difficult
Snapshots supported No Yes Yes
Performance Slightly higher than DIR-Qcow2 Slightly lower than DIR-RAW Slightly higher than DIR-RAW

Read more about performance of different local storages in the article "Comparison of local storage performance".

Network storage

iSCSI, NFS, network LVM, RBD, and GlusterFS are used as network storages. RBD and GlusterFS can be added to the group of the so-called fault tolerant network storages. When a new network storage template is created, access parameters are to be specified, such as hostname, IP address of the server, or directory of virtual disk images, depending on the type of the network storage.


Remote server is used as storage. Access to this remote server is established over iSCSI, the block level protocol. The iSCSI protocol describes transfer of SCSI packages via the stack of TCP/IP protocols.

SCSI (the Small Computer Systems Interface) is a set of protocols allowing to work with input/output devices, especially with data storage systems. SCSI has the client-server architecture. Clients (initiators) execute SCSI commands to create requests to components (logical units) of servers which are called targets, or target devices.

VMmanager uses Open-iSCSI, one of the versions of iSCSI protocol for GNU/Linux.

iSCSI documentation is available on the website

iSCSI storage settings are described in our documentation: VMmanager: ISCSI storage.


In this case, remote server is used for storage. Access to this server is established over NFS (the Network File System), the file level protocol.

NFS utilizes the system of remote procedure calling - ONC RPC (Open Network Computing Remote Procedure Call). This system is based on RPC (Procedure Call Model), the protocol of remote procedure calling.

The model of remote procedure calling can be described in the following way. The control flow connects the process of calling of the server from the client and the server process. The client process sends a message with the call and the request to the server process, and then it waits for the return message. The request includes procedure parameters, while the return message includes the results of the procedure. After the return message has been received, the procedure results are extracted and sent to the client. The process on the server side becomes inactive and awaits the call. During calling, the server process extracts procedure parameters, calculates the results, sends the return message, and then waits for the next call.

Transfer in ONC RPC is performed via the protocols TCP, UDP, and XDR.

The current version of NFS is NFSv4.

NFS documentation is available on the website

You can find more information about installation and setting of the NFS storage in our documentation NFS storage.

Network LVM storage

The network LVM storage is another version of the standard LVM. Here, the physical volume with a group of volumes is the network device.

Access to the remote server is extablished over the iSCSI protocol.

Installation and setting of the network LVM storage is described in our documentation Network LVM-storage.

Fault tolerant network storage

Ceph and GlusterFS are distributed file systems allowing to create a scalable cluster of servers with different roles, for storage or data replication purposes and load distribution, which guarantees high availability and reliability. If one of the disks, nodes, or a group of nodes fail, Ceph and GlusterFS save the data and restore the lost copies on the other available nodes automatically, until the failed nodes or disks are replaced. The cluster stays online without a second downtime or any issues for the customers.

Ceph RBD

Ceph gives a few options for access to data: block device, file system, or object storage. VMmanager supports RBD, the distributed block device with kernel client and QEMU/KVM driver. With RBD, virtual disks are distributed among several objects and stored in the distributed Ceph (RADOS) storage. Functionality and operation of a Ceph cluster is supported by utilities installed above the operating system, such as the following:

  • Monitors (MON): Ceph-mon supports cluster status cards, including MON, MGR, OSD, and CRUSH cards. The given cards are required to coordinate Ceph utilities with each other. Also Ceph-mon is responsible for authentification between utilities and clients. We recommend using at least 3 MON;
  • Managers (MGR): Ceph-mgr is responsible for tracking execution time and current status of the Ceph cluster, including storage usage, ongoing operation data and system load. Also, Ceph-mgr is utilized to install python plugins to represent and manage a Ceph cluster. We recommend using at least 2 MGR;
  • Object Storage Daemon (OSD): Ceph-osd is responsible for storage, replication, restoration and rebalancing of data. It works with MON and MGR by giving them information about presence or absence of other OSD. We recommend using at least 3 OSD;
  • Metadata Server (MDS): ceph-mds provides storage of Ceph Filesystem metadata. Please note that MDS is not needed for Ceph Block Devices or Ceph Object Storage.

You can use one server with two roles in small clusters, e.g. for storage or as a monitor. In large-scale clusters, it is recommended to launch utilities on different machines.

A VMmanager node can be used as a Ceph cluster node. However, we cannot recommend it since it can lead to high server load, or to storage or VMmanager node restoration, if such server goes out of service.

RBD storage supports only the RAW format of virtual disks.

You can find Ceph documentation on the website

You need to do the following in order to set up storage of virtual machines in the distributed Ceph storage with RBD:


With GlusterFS, virtual disks are distributed over a few objects and are stored in the distributed storage this way. The distributed storage consists of volumes which are physically located on different servers. There are a few ways to record data to the volumes:

  • Distributed: data is distributed evenly across all servers without duplication. Such volumes are not protected by GlusterFS. If one of the servers or its disks fails, its data will not be available.
  • Replicated: data is recorded at least on two servers.
  • Striped: the incoming data is divided into parts which are recorded to different servers. When a request comes, data is written off servers recursively. If one of the servers or its disks goes out of service, the volume will be out until the failed node is restored again.
  • Distributed Striped: data is distributed evenly between subvolumes. Inside each subvolume, data is divided into parts and recorded to different servers parallelly.
  • Distributed Replicated: data is distributed evenly between subvolumes. Inside each subvolume, data is recorded at least to two servers. This option is the most reliable.

GlusterFS does not need the centralized server for metadata.

GlusterFS supports both RAW and Qcow2.

Find documentation on GlusterFS on the website.

You can find more information about GlusterFS settings in our documentation "GlusterFS storage".