Archive tier
QoreStor's archive tier feature enables QoreStor data to be quickly and easily archived to long-term Amazon S3 Glacier or Amazon S3 Glacier Deep Archive storage. Using your existing data management applications (DMAs) and a supported protocol (Object(S3), VTL or RDS), files can be written to a QoreStor container and migrated to your archive tier according to easily defined policies. QoreStor provides a policy engine that allows you to set file age and on-premises retention criteria to be used in identifying which files are most suited for replication to the cloud. Policies are defined at the container level and apply to all files within that container.
Using the QoreStor Cloud Policy, you can replicate files based on:
- Idle time - replicate stable files idle for more than the selected number of hours.
- File extensions - replicate files that match or do not match names in a list of extensions.
- Regular expressions - include or exclude files based on their match to configured regular expressions.
- File locations - replicated files in a list of directories, or all files except those in a list of directories.
Any data that is archived from the QoreStor instance by the archive tier is encrypted with zero knowledge encryption. The encryption keys are solely owned by you. If the encryption keys are placed in the archive tier, a passphrase is used to encrypt those keys and that passphrase is only known to you. For added security, QoreStor obfuscates metadata names such as blockmap and data store objects that are stored in the archive tier.
Data stored in the archive tier is not available for immediate recovery. When a recovery is initiated, the data stays in the archive tier while a copy is made in S3 standard storage and kept for an amount of time specified by the archive_retention_in_warm parameter. Although recovery times may vary, the general expectations for recovery times are:
- Amazon S3 Glacier storage: 3-5 hours
- Amazon S3 Glacier Deep Archive: within 12 hours
Object interface (S3-compatible)
QoreStor's Object container provides an object storage interface which enables customers to write Object data(S3 format) directly to Qore. This allows solutions that leverage an S3-based connection to send data directly to a QoreStor instance instead of Amazon S3 with the added benefits of deduplication, encryption, replication and network optimized data transfer.
With object storage enabled, QoreStor creates a ObjectStorageGroup storage group and ObjectContainer container. This container can be configured with a cloud tiering policy to seamlessly move data to long-term storage. QoreStor allows you to configure multiple buckets within your object container namespace, each with different locking and retention settings.
Object Direct Storage
Object storage organizes files and their associated metadata into objects, which are then stored in a flat address space. Object storage provides greater insight to data usage, improved scalability, and lower storage costs when compared to traditional file or block storage. QoreStor can be installed in object-direct mode, which utilizes object storage for the main data repository, while metadata is written to higher performing storage. For more information, refer to the topics QoreStor Installation Modes and Installing QoreStor with Object Direct Storage in the QoreStor Installation Guide.
|
|
NOTE: When QoreStor is installed in an Object Direct configuration, if the backend object storage is not accessible QoreStor will go into Manual Intervention mode. |
Disaster Recovery from the cloud
When configured with a Cloud Tier or Archive Tier, QoreStor provides the ability to capture QoreStor configuration information as well as storage group and container data. With this information, it is possible to re-create a lost or failed QoreStor server using the data in the cloud.
The disaster recovery process utilizes the QoreStor DRSNAP file, which is a disaster recovery reference file stored in the cloud and encrypted with the same passphrase used to encrypt the encryption keys for the cloud tier. The DRSNAP file is created when a cloud tier is created, and is updated each time a container is created or when the cloud tier passphrase is changed. The DRSNAP file contains:
- Configuration information (storage groups, containers, connections and VTL data).
- Cloud or Archive Tier and cloud containers for the source containers.
- Name space for the source containers and the cloud containers.
- Cloud container blockmaps.
- Source containers files are stubs and point to corresponding Cloud container blockmaps.
- Datastore headers for Cloud or Archive Tier.
- Dictionary for the Cloud or Archive Tier is repopulated.
In the event of QoreStor server failure, a recovery can be initiated on a new, licensed QoreStor server to restore the previous configuration. At a high-level, a recovery will go through the following steps:
- Connect to your cloud provider with configured credentials and passphrase
|
|
IMPORTANT: The required passphrase is the passphrase used when created the cloud tier. Without the cloud tier passphrase, recovery is impossible. |
- Restore the archived DRSNAP file to the new QoreStor server and decrypt the file.
- Parse the DRSNAP file for cloud-replicated containers as well as other storage groups and containers.
- Rebuilds the cloud tier using information collected.
Refer to Performing a recovery from the cloud for more information.
When performing a data recovery from an archive tier, you must first restore all datastores to standard AWS S3 storage using the AWS Management Console. Refer to Manually restoring datastores from Amazon S3 Glacier and Restoring from archive tier for more information.
