Backup process

From ISPWiki
Jump to: navigation, search

This article describes how backups are made in ISPmanager.

See also:

Collecting data

Before starting a backup process, data from the control panel's modules that support backups are collected. This process is started by calling the backup.list function for which the corresponding modules are set as events. The function returns the following XML-document:

<doc>
 
  <module name="user">
 
    <type name="user" filter_files="on"/>
 
  </module>
 
  <elem>
 
    <backup>
 
      <type>user</type>
 
      <name>user</name>
 
      <partname>metadata</partname>
 
      <filter_files/>
 
      <separate>on</separate>
 
      <path>/var/www/user/data</path>
 
    </backup>
 
    <data>...</data>
 
  </elem>
 
  <module name="db" owner="user">
 
    <type name="db"/>
 
    <type name="db.users"/>
 
  </module>
 
  <elem>
 
    <backup>
 
      <type>db</type>
 
      <subtype>mysql</subtype>
 
      <name>db</name>
 
      <owner>user</owner>
 
      <prepfunc>db.dump</prepfunc>
 
      <rootarchiver>on</rootarchiver>
 
      <edit_elid>db->MySQL</edit_elid>
 
      <depend type="user" name="user"/>
 
    </backup>
 
    <data>...</data>
 
  </elem>
 
</doc>

The module node

The module describes a certain module. Filters are configured per each module. The @name attribute - module name @owner - parent module. Depending on the modules' hierarchy, child dependencies are identified while restoring data. The type node included into the module associates data types with certain module.

The elem node

The elem node describes data or their part. <elem> contains two chield nodes: <backup> and .

  • <backup> - contains information and management elements that are required for backup system. The <backup> node is described in more details below.
  • <data> - certain information that you should save for future data restore. Contents of this node is specified by the user who is making a backup copy of that module. Normally the module contains nodes from the data creation/edit form, where the node name is parameter name, node contents - parameter value.

<backup>

This node is the interface for a general backup system and a specific function for saving certain data. This module contains elements from the set specified earlier:

  • type (required) - type, such as webdomain, user,etc.
  • subtype (optional) - subtype, which is used for databases only. Example: mysql, posgresql. The subtype is used together with the type dot separated (".").
  • name (required) - name. For example: the name for "user" will be "username".
  • owner (optional) - Owner - owner of that data. This is a mandatory field for data with owners.
  • path (optional) - Absolute path to the object. For user - path to the home directory.
  • partname (optional) - If the object is specified in different directories, like, for example, user - home directory and panel settings, for each directory you may specify additional elem with the same type, subtype, name, owner, but with a different path. To deffirintiate elem and specify correct recovery order, you should specify a separate partname for each elem.
  • display_name (optional) - name that will be displayed in the panel interface.

Note the combination of type, subtype, name, partname and owner must be unique withing the list backup.list.

  • prepfunc (optional) - object preparation function. db.dump for a database. If the data filter is set, but databases are still backed up, the db.dump function will be called for creating databse dump and will return the XML containgin path to the damp in the <path> node. Nodes and contents as their parameters and values will sent to stdin for db.dump.
  • filter_files (optional) - files located in path can be filtered while filtering files.
  • view_files (optional) - files located in path can be viewed and selectively restored.
  • delete (optional) - delete object files upon backup. It can be used for database dumps, temporary metadata and parts of configuration files. Values specified in path will be deleted. If prepfunc is specified, it will be deleted as it is.
  • force (optional) - force save. Ignore incremental, differential and other backup types for files of that object.
  • depend (optional) - objects, on which this object depends. There can be several nodes. The node has the following attributes: @type - type of the object on which depends, @name - name of the object, on which it depends. The depended object will be searched by the same owner that this object has.
  • rootarchiver (optional) - when creating this node, the object files for which the owner is specified, will be archived with superuser permissions. Otherwise, they will be archived with permissions of the user specified in the owner field.

The list will be divided into elem, each elem processes separately.

Choosing data for backup

The filtering rules that were added in the Backup plans module will be applied for each element. The filters will be applied as they appear on the list. Once the last filter is applied and elem is "active", the item will be added into the backup copy; otherwise, it will be deleted.

Note You may perform all the steps below only if elem were filtered successfully.

Preparing data

For each elem with prepfunc specified the corresponding function will be called. Contents of /elem/backup and /elem/data will be sent to stdin, where node names are parameter names, node contents - parameter values. Parameters values from /elem/data will clear the corresponding values from /elem/backup

The resulting XML contains either the path node containing the path to files, or the data node containing metainformation for recovery.

VMmanager and VEmanager do not return a path to their virtual machine. They start a new background task to prepare the virtual machine. Then they call the function backup.continue or backup.break if errors occurred.

Applying user filters to files

For each elem with the specified path a list of files for backup will be generated. If the path is a file, the list will contain one file only. If the path is a directory, recursive walk through a directory tree will be made. File filters will be applied for file and idrectories names while walking. Sizes, modification dates and file permissions will be saved.

Backup types

For each element, files are filtered depending on a backup type specified in backup plan edit form.

Full

When making a full backup copy the list of files will remain unchanged.

Incremental

A full backup copy will be made and the list of files will remain unchanged. If the number of days from the last full backup exceeds the value specified in the backup plan settings, a full backup copy will be made and list of files will remain unchanged. Otherwise, an incremental copy will be made - the list will contain the files that were modified since the last backup copy (full or incremental). The file is considered modified, if its size and/or modification date were changed.

Differential

A full backup copy will be made and the list of files will remain unchanged. If the number of days from the last full backup exceeds the value specified in the backup plan settings, a full backup copy will be made and list of files will remain unchanged. Otherwise, a differential copy will be made - the list will contain only those files that were modified since the last full backup. The file is considered modified, if its size ans/or modification date were changed.

Archiving

A new list of files is generated based on the list of files and directories. This list is sent to the tar archiver through the -T option. If the owner node is specified and the rootarchive node is not specified, tar will start with permissions of the user specified in the owner node, otherwise - with the control pane'l permissions. The archive will be compressed with the gzip utility. Tar is started with the key --ignore-failed-read. Tar results are checked for errors and missing files.

Saving archive into storage

A path to save the archive and the lst file is generated as follows:

The directory backup_{id}, where {id} - is a backup date in the format 2014-08-25_16-45-23.

If the data belong to user, the user directory is added.

A directory with the data type name (webdomain, db) is added.


A file name is generated basing on the following template: {datatype}-{dataname}-{owner}-{index}.

  • {datatype} - data type (user, admin, webdomain, db)
  • {dataname} - data name. The Latin alphabet letters can be used. The following symbols cannot be used ( ;, <, >, |, &, \, ', \t, `, *, ?, /, :, " )
  • {index} - order number of the current archive