Chat now with support
Chat with Support

QoreStor 7.0.0 - User Guide

Introducing QoreStor Setting up your QoreStor system Configuring QoreStor settings Managing containers Managing local storage Managing cloud storage Managing replications Managing Users Managing QoreStor Remotely Monitoring the QoreStor system Support, maintenance, and troubleshooting

Restoring from archive tier

Depending on the container type, data is send to the archive tier by different methods. For RDA and Object containers, data is archived based on the Archive Tiering Policy. For VTL containers, exporting the cartridge from the backup application will trigger the movement of cart data to cloud. Restoring data from an archive tier differs in some ways from a standard restore process. When restoring from an archive tier, no files are saved to on-prem storage. Instead, files are copied from the archive storage (Glacier or Amazon S3 Glacier Deep Archive) to warm AWS S3 storage for a period of time specified by the Archive Retention in Warm Cloud setting.  When restoring from an archive tier, consider the following:

  • Restoring from archive storage is a two-step operation. First, archive data is restored to standard AWS S3 storage, then the objects are read from there.
  • Restoring from Glacier or Amazon S3 Glacier Deep Archive storage takes significantly longer than standard restores. No notification is issued when restored objects are available. You may view the status of restore operations in the AWS Console. Refer to the Amazon S3 document Checking Archive Restore Status and Expiration Date for more information. To perform a batch restore for disaster recovery purposes, refer to Manually restoring datastores from Amazon S3 Glacier.
  • When restoring objects from archive, you are charged for both the archive copy and the restore copy in warm storage. Use the Archive Retention in Warm Cloud value to minimize the duration objects are kept in warm storage.
  • For restoring VTL cartridge data, the command vtl --import must be run on the QoreStor server.

Manually restoring datastores from Amazon S3 Glacier

To perform a disaster recover from an archive tier, you must first restore all datastores to standard Amazon S3 storage. The procedure below will guide you in performing this action in the AWS Console.

  1. Using the ls command, recursively get the contents of the cds folder in the cloud bucket you wish to restore.

    NOTE: Note that "bucket_name" is the bucket name. Replace this with your bucket name

    # aws s3 ls s3://bucket_name/cds --recursive | awk '{print "bucket_name,"$NF}' > manifest.csv

    This command will create a manifest.csv file. A sample version of a manifest.csv file is shown below:

    	
    # head manifest.csv
    bucket_name,cds/0000/0000/0000/0001/158/158.imap
    bucket_name,cds/0000/0000/0000/0001/158/158_0
    bucket_name,cds/0000/0000/0000/0003/357/357.imap
    bucket_name,cds/0000/0000/0000/0003/357/357_0
    bucket_name,cds/0000/0000/0000/0004/439/439.imap
    bucket_name,cds/0000/0000/0000/0004/439/439_0
    bucket_name,cds/0000/0000/0000/0008/869/869.imap
    bucket_name,cds/0000/0000/0000/0008/869/869_0
    bucket_name,cds/0000/0000/0000/0010/1035/1035.imap
    bucket_name,cds/0000/0000/0000/0010/1035/1035_0
    		 
  2. To ensure the manifest file can be uploaded successfully, add an MD5 hash.
    # openssl md5 -binary manifest.csv| base64
    BBpU3xTS3/XKnmDm+uqdng==
  3. Paste the MD5 in the put-object command. The resulting ETAG can be used for initiating the restore job.

    NOTE: .--key <path> can be used for uploading manifest into any prefixed location in that bucket.

    # aws s3api put-object --bucket bucket_name --key manifest.csv --body manifest.csv --content-md5 BBpU3xTS3/XKnmDm+uqdng==
    {
    "ETag": "\"041a54df14d2dff5ca9e60e6faea9d9e\""
    }
  4. The uploaded manifest, it's location and ETAG can be used for initiating a batch operation restore job through the AWS console. Alternatively, you may use the create-job API /'as follows:
    # aws s3control create-job --no-confirmation-required  --account-id  177436582181 \
    --operation '{"S3InitiateRestoreObject" : { "ExpirationInDays" : 7, "GlacierJobTier" : "STANDARD" }}' \
    --report '{ "Bucket":"arn:aws:s3:::bucket_name",\
    "Prefix":"batch", \
    "Format":"Report_CSV_20180820", \
    "Enabled":true,\
    "ReportScope":"AllTasks" }' \
    --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},\
    "Location":{"ObjectArn":"arn:aws:s3:::bucket_name/manifest.csv","ETag":"\"041a54df14d2dff5ca9e60e6faea9d9e\""}}' \
    --role-arn arn:aws:iam::177436582181:role/BatchOperationUser \
    --priority 10
    {
    "JobId": "ea4915fc-2cbb-4d00-895e-a363743b9c8c"
    }
  5. Restored objects will be ready for readback after 4-6 hours for Amazon S3 Glacier (10-12 hours for Amazon S3 Glacier Deep Archive). You can click on an individual CDS object to make sure it is ready for readback (ongoing-request = false, expiry-date = <some future date>) before attempting to restore from QoreStor. Optionally, you can perform a head-object inquiry on any one of the CDS entries from the uploaded manifest file:
    # aws s3api head-object --bucket bucket_name --key cds/0000/0000/0000/0001/158/158_0
    {
    "AcceptRanges": "bytes",
    "Restore": "ongoing-request=\"false\", expiry-date=\"Sat, 05 Sep 2020 00:00:00 GMT\"",
    "LastModified": "2020-09-03T08:20:42+00:00",
    "ContentLength": 4245203,
    "ETag": "\"98b38ff76a2dc04f423ba954af44c052\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "StorageClass": "GLACIER"	

    NOTE: Note that the time required to run the ls command can be quite long for a high number of objects. If the command fails due network or connection issues, please restart the operation.

    # time aws s3 ls s3://st-6300-19-archive/cds --recursive | awk '{print "st-6300-19-archive,"$NF}' > st-6300-19-archive_manifest.csv
    real    43m24.906s
    user    39m1.575ssys     1m11.745s
    
    # ls -al st-6300-19-archive_manifest.csv
    -rw-rw-r-- 1 jayant jayant 163138124 Sep  3 05:15 st-6300-19-archive_manifest.csv
    
    # wc -l st-6300-19-archive_manifest.csv
    2680848 st-6300-19-archive_manifest.csv
    
    # time aws s3api put-object --bucket bucket_name --key st_manifest.csv --body ./st-6300-19-archive_manifest.csv --content-md5  ni/tJMLCpB0l9A1m3Te3sg==
    {
    "ETag": "\"9e2fed24c2c2a41d25f40d66dd37b7b2\""
    }
    
    real    0m5.281s
    user    0m1.930s
    sys     0m0.560s

Performing a recovery from the cloud

To recover your QoreStor configuration and cloud-replicated data from the cloud, perform the steps below. Before peforming these steps, make sure you have the following:

To perform a disaster recovery

  1. On your QoreStor server, execute the recovery command
    maintenance --disaster_recovery --cloud_string <name> --container_name <name> --cloud_provider_type <name> --passphrase <name> [--logfile <name>]

    where

    --cloud_string          cloud connecion string.
    --container_name        cloud cotainer name.
    --cloud_provider_type   cloud provider type.
    --passphrase            passphrase.
    --logfile               log file path.

    This will regenerate configuration data, initialize the QoreStor dictionary, and configure container namespace and blockmaps. When completed, you will see the message

    Processing Datastores: Done	
  2. After the data recovery process is complete, perform a filesystem repair with the command
    maintenance --filesystem --repair_now

    When the file system repair is finished, the process is complete.

Next steps

Depending on your configuration, there may be several steps required after recovering your QoreStor server. Some actions to consider are:

  • If you are using QoreStor with NetVault Backup, you will need to add the new QoreStor as target device and add the container.
  • Depnding on your DMA, you may need to reconfigure DMA or client connections to reference the new QoreStor server.
  • Once a disaster recovery completes, the recovered source containers will be unencrypted. Before ingesting new data into the recovered containers, you must enabled encryption on the recovered storage groups.

    NOTE:The recovered containers contain only stub files. The data remains encrypted in the cloud tier.

 

 

Managing replications

In the QoreStor GUI, you can set up and manage data replication operations. Such replication operations include, creating new replication relationships, managing or deleting existing replication relationships, starting and stopping replication, and displaying current replication statistics.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating