FMS HA Server Restarter Problems as for mysql backup locking the database tables (77586)

The nightly mysql backup is causing the cluster to failover

The backup is done using mysqldump command to backup: mysqldump --user=$USERNAME --password=$PASSWORD --host=$DBHOST --quote-names –opt

As a result the procedure locks the tables and Foglight server is not be able to write to the database the Foglight management server will queue up the transactions but will not be able to persist the data to the backend that will stop the FMS from responding to will initiate a failover

When a server running in HA mode cannot establish connections to the database, the re-starter will restart it. The reason is because the server becomes unresponsive if the database becomes unresponsive.

If the database is not reachable, Foglight management server is able to recover when the DB comes back, however if the DB is reachable but unresponsive, it causes the FMS also become unresponsive.

In an HA setup, this causes the re-starter to kill the server.

Foglight requires a functional DB If backing up of the DB makes it unavailable for Foglight to use then we recommend to use a “Hot backup” tool which is capable of backing up a running database.

One such tool is InnoDB Hot Backup.

For more information please refer to http://www.innodb.com/wp/products/hot-backup