Incremental archive to S3 bucket with Glacier lifecycle policy enabled fails (4038370)

Incremental archive to S3 bucket with Glacier lifecycle policy enabled fails

This article describes how the archive to Amazon Glacier function works in Rapid Recovery 6.2.0 and newer and why previously scheduled incremental archives to S3, with a bucket lifecycle policy set to move data to Glacier, fail after upgrade to 6.2.0 and newer.

The logic for validating the recovery points contained in an archive prior to performing an incremental archive changed in Rapid Recovery 6.2.0. The change was made to ensure that there are no gaps in the recovery point chain which would make the recovery points in the incremental archive useless. This validation change is not compatible with archives to S3 buckets that have a lifecycle policy set to move all data to Glacier since it requires that the Rapid Recovery core be able to read the metadata of the archive and any data stored in Glacier is not readable. 

• How the "Use Glacier" function works when performing archives:
• Select the "use glacier" checkbox in the archive process and choose your desired S3 bucket
• Ensure there are no lifecycle policy rules set on the S3 bucket you have chosen for the archive. If the bucket is moved to Glacier, the job will start failing immediately after the move.
• Data and metadata will be written to the S3 bucket chosen.
• The datafile.bin file will be marked for storage in glacier and will be moved to glacier within 24-48 hours by a background AWS job. Rapid Recovery has not control over how quickly data is moved to Glacier.
• Once moved, the datafile.bin file cannot be seen in the S3 bucket
• To view the datafile.bin you need to connect to Glacier with an S3 file browsing utility such as S3 Browser and run a "get inventory" command. You must wait for a few hours while that command runs and populates the list of files in that Glacier Vault (every action in Glacier takes a long time to complete which is why the storage is so cheap)
• Scheduled incremental archives are compatible with the "use Glacier" archive feature since the core uses the metadata in S3 to validate the contents of previous archives and then creates a new incremental archive in the S3 bucket and marks the datafile.bin for move to Glacier. This allows incremental updates of your archives while using Glacier to store the data.

 

Unfortunately, all scheduled incremental archives written with versions of Rapid Recovery 6.1.3 and older and then moved to Glacier with an AWS Bucket Lifecycle Policy will fail after upgrade to 6.2.0. Rapid Recovery 6.2.0 implements logic to better validate the data within archives before performing an incremental update of the archive. To do this the Rapid Recovery core must read the metadata in the archive and when a bucket has been moved to Glacier, it is no longer available for read and so the Rapid Recovery core receives an error from Glacier when trying to access the needed metadata.

There is no upgrade path for scheduled incremental archive jobs stored in Glacier with versions of Rapid Recovery 6.1.3 and older. Our recommendation is to create a new scheduled incremental archive using the "use Glacier" option (available in RR 6.2.0 and newer) and begin archiving your data to an S3 bucket without a Lifecycle Policy configured.