Foglight Experience Monitor 5.8.1 - Installation and Administration Guide

Installing an upgrade

Download upgrades from the Quest Support Portal and install them using the following procedure. This method of upgrading is recommended.

IMPORTANT: The Log page is displayed until the upgrade installation process requires all system applications to stop. Consequently, the web console will not be available and the system may be offline for a period of 30 minutes to 3 hours. Do not reboot the appliance during the upgrade. When the console program is available during the database upgrade process, you can log in to see the progress of the upgrade. When the web console login page becomes available, the upgrade is complete.

NOTE: The upgrade file must have a .pkg extension, otherwise you will receive an error message.

Backing up and restoring data

Backups include all databases and configuration settings that are needed to recreate an identical system on a new appliance. Settings such as the appliance's IP address and gateway IP are not included in the backup and must be reconfigured manually using the console program.

The appliance’s backup and restore functions can utilize either File Transfer Protocol (FTP) or Secure Shell (SSH) as a network protocol to transfer files across the network to and from a backup server that you have designated. For environments with stringent security requirements, it is recommended that you utilize SSH as the backup protocol.

Backups can be scheduled to occur on a daily or weekly basis or they can be initiated manually. You can also manually initiate a backup of the configuration files and databases which is useful for quickly transferring configuration from one appliance to another.

All backups can be configured, scheduled, and initiated on the Backup & Restore page, accessed by clicking Configure > Appliance > Backup & Restore.

IMPORTANT: Most system processes are halted during the backup, but the appliance continues to collect metrics. Metrics collected during the backup are queued and then loaded into the database once the backup has completed and all of the system processes have restarted.

To ensure that the backups for your appliances are secure and reliable, it is essential that you use a backup server with at least 20 GB (gigabytes) of available disk space. Your backup server should also have a fast disk I/O subsystem that is capable of writing large files and has a transfer rate that is able to keep up with the demands placed on it.

For best performance, use a backup server that is not being heavily loaded by other unrelated applications. If your backup server is heavily loaded with other activity, it is possible that backups from your appliances could experience timeouts which would cause them to fail.

It is also possible that backups can take many hours to complete. In general, a backup can take approximately two to three hours but this amount of time can increase if the backup server is heavily loaded and responding slowly. During backups, the appliance's web console is unavailable.

To ensure backup availability in the event of a system failure, it is highly recommended that you schedule a daily or weekly backup. It is recommended to avoid scheduling backups at midnight due to the fact that the appliance conducts regular database maintenance functions at midnight every day. This could increase the amount of time that the backup requires.

In addition, if you have multiple appliances using the same backup server, it is important that you do not schedule the same backup time for more than one appliance. Doing so can cause your backup server to become overloaded, triggering timeouts which will result in incomplete backups.

The backup server can be running any operating system as long as it supports an FTP or SSH server.

Foglight Experience Monitor supports two protocols for backups: FTP and SSL. The following steps explain how to set up a backup server to use FTP.

NOTE: Make sure that you do not configure the FTP server solely for anonymous access, and that you do not disable write access.

NOTE: You must enter an IP address not a domain name.

This procedure describes how to set up backups using SSL as the communication protocol. This method provides a more secure and reliable approach than FTP.

NOTE: A popular open-source SSH server that works on Windows® platforms is cygwin which you can download here: http://www.cygwin.com/

IMPORTANT: Permission settings are required.

TIP: You do not need to enter a password.

The verification process transfers a small file to the backup server and then retrieves it. If the file retrieved matches the file that was transferred, then the backup server is verified. A rough speed calculation is also performed.

Download and upload times recorded during the verification process are displayed in a window. These timings vary each time you run the verification process because of differences in network utilization and CPU and disk utilization on the appliance. Generally, you should expect to see timings greater than one MBps.

If your timings are lower, backups could take a very long time to complete. Investigate the quality of the network path between the appliance and the backup server and also the performance of the backup server itself. You may need to secure a backup server with higher performance or relocate the backup server so that the backups have a shorter and faster path across the network.

If the verification fails, you need to check out the items in the following list of possible causes:

The complete backup includes data from the metric database and the configuration database.

IMPORTANT: A backup can take as long as five hours if the database is large. The amount of time required by the backup process is dependent on how much data needs to be written to the backup server and how often you conduct backups.

For more information, see these topics:

Restoring a backup

Restoring a backup is normally only necessary when a catastrophic hardware failure requires replacement of your appliance. Follow the steps below to restore a backup.

IMPORTANT: A backup can only be restored on an appliance that has the same version number as the appliance where the backup was created.

IMPORTANT: If you are restoring a backup from a different appliance, any SSL keys that you have configured for the monitored servers is no longer usable by the appliance. For security reasons, SSL keys are encrypted with an algorithm that uses characteristics of the hardware in the appliance itself. Therefore, the SSL keys cannot be decrypted on the new appliance because they were already encrypted on a different machine. You need to reload all SSL keys through the user interface. If you are restoring a backup to a machine with a different IP address all settings, such as the list of monitored servers and protocols, are carried over.

How backup works

Backups begin at their scheduled time or whenever you initiate one manually as described in Manually starting a complete backup. At the beginning of a backup all non-essential system tasks are shut down. This includes the Apache™ web server and all database processes. This is required in order to ensure that no changes are made to any of the databases while a backup is underway.

The appliance continues monitoring traffic and collecting metrics during the backup process. Data files containing new metrics are queued up and are loaded into the database when the backup has completed.

When the first backup for an appliance is initiated, the system creates a unique directory for the appliance on the backup server. The file name of this directory is based on the MAC address of the control port (eth0) of the appliance. For example, if the MAC address of eth0 is 00:30 is named 0030482DF610_backup. This directory only contains backups from a single appliance.

Within this top-level directory, additional directories are created containing a timestamp for the date and time the backup was created. For example, the directory 0030482DF610_backup/20080107164500Z is created for a backup that was initiated on Jan. 7, 2008 at 16:45 UTC. Within that directory, multiple subdirectories are created for individual databases and configuration files. You should never move or modify any of these files contained in the directories. Doing so causes the backup to become unusable.

Backup directories contain three types of files: TGZ, MD5, and XML. The TGZ files are compressed tarballs of database or configuration files. The MD5 files are MD5 hashes of the corresponding TGZ files. These are used to verify that the TGZ files are not corrupt. You can verify the integrity of your backup by computing the MD5 hash of each file and comparing that to the MD5 key files that are created by the backup process.

The XML file, backupinfo.xml, gets created in each backup directory. This file contains information about each component of the backup. It is used by the system for conducting subsequent backups and for restoring.

NOTE: Some of the directories referenced in this file could be from previous backups due to the incremental nature of the backup process (described below).

Backups include all configuration settings and metrics with the exception of the following:

As outlined in the previous section, backups can be configured to run on a daily or weekly basis on the Configure > Appliance > Backup & Restore page.

A full backup can require disk space equaling 20% of the total size of the Foglight Experience Monitor database partition. To find the current size of the database partition, click Help > System Information and look for Database Partition Size.

An incremental backup can require up to 20% of the disk space consumed on the database partition since the last backup. To monitor your database partition usage over time, use the Configure > Appliance > Database page.

For example, let's consider a database partition size of 3.6 TB. The initial backup of this database requires approximately 720 GB (about 20% of the partition size). If the system collects about 50 GB of new data a week, you also need an additional 10 GB for each subsequent incremental backup. Therefore, the disk space required to hold the initial backup and ten weeks of incremental weekly backups is 820 GB, calculated as follows:

720 GB + (10 GB * 10) = 820 GB

The disk space requirements continue to increase over time until these backups are cleared off the backup server. At that point, the initial backup again requires 720 GB and the same estimates for incremental backups apply.

It is recommended that you back up your appliance on a regular basis whether it be daily or weekly. It is also recommended that approximately once a month, you archive the contents of the directory server off to tape, DVD, or CD, on your backup. Doing this keeps your FTP server storage requirements to a minimum and allows for offsite archival of your appliance data.