How to Restore Folders from Glacier to S3

Amazon S3 is a cloud storage provided by AWS. You can store a bulk amount of data in S3 for a relatively low price. But if you are not accessing those data so often, it is better to move them to Amazon Glacier. So, how this Amazon Glacier is different from Amazon S3? The Pricing of storage in Amazon S3 is $0.023 per GB wherein Amazon Glacier, it will cost only $0.004 per GB. That’s a big difference. Right? So let’s get started on how to restore folders from Glacier to S3.
You can add a life cycle policy for an S3 bucket in such a way that, after a period of time the files will be automatically moved to Amazon Glacier. So, you will not need to worry about moving each individual files to Amazon Glacier after a period of time. You can follow the Amazon documentation for creating the life cycle. https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html
Objects stored in Glacier are not immediately accessible. For accessing those objects, you will have to restore a temporary copy of it to its S3 bucket for the duration (number of days) that you specify. After restoring those objects, you can download it. If the data in the Glacier are few, you can directly restore it from the S3 console itself. You can follow the Amazon documentation for this. https://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html. But there is a drawback to this. In S3 console, there is no option to select an entire directory. You will have to select each file one at a time. So, if there is a huge amount of files, it will be very difficult and time-consuming to select each file. The best option is to use the AWS CLI to restore those files. The AWS CLI is a unified tool to manage your AWS services from a terminal session on your own client. If you are using a Linux machine, you can install AWS CLI by following the doc: https://docs.aws.amazon.com/cli/latest/userguide/install-linux-al2017.html and if you are using a Windows machine, you can install AWS CLI by following the doc:
https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html
Once you have installed AWSCLI, you will need to configure AWSCLI by using the following command.
aws configure
Here, the AWS CLI prompts you for four pieces of information and you will need to enter the details such as AWS Access Key ID, AWS Secret Access Key, Default region name and Default output format. You can find more details by following the doc: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
After configuring the AWSCLI, enter the following command which will list all the objects under Glacier. Here you will have to replace the BUCKETNAME with your bucket name and PREFIXNAME with the prefix.

aws s3api list-objects-v2 –bucket BUCKETNAME –prefix PREFIXNAME –query “Contents[?StorageClass==’GLACIER’]” –output text | awk ‘{print substr($0, index($0, $2))}’ | awk ‘{NF-=3};3’ > file.txt

Let us deal this with an example. Here I want to restore all the folders and files from a particular directory of my bucket. Let us take the directory location as: s3://remote-server22/all-accounts/globalel/snap.1  Here, “remote-server22” is my bucket name and “all-accounts/globalel/snap.1” is my prefix name. So, the command should be like:

aws s3api list-objects-v2 –bucket remote-server22 –prefix all-accounts/globalel/snap.1 –query “Contents[?StorageClass==’GLACIER’]” –output text | awk ‘{print $2}’ > file.txt

Here, we will get a list of all the objects under Glacier rules in the location “s3://remote-server22/all-accounts/globalel/snap.1” to a text file named “file.txt”.
Then create the following script in your machine to restore the contents in the file “file.txt”.

#!bin/sh
for i in $(cat file.txt); do
echo “Start restoring the file $x”
aws s3api restore-object –restore-request Days=1 –bucket remote-server22 –key “$x”
echo “Completed restoring the file $x”
done

Here, I have mentioned the day as 1. So, I will get a temporary copy of all the Glacier objects for the duration specified in the restore request. AWS will take almost 3-5 hour to complete the restoration. After this time interval, run the following command to download those files to your local machine.

aws s3 sync s3://remote-server22/all-accounts/globalel/snap.1 /backup –force-glacier-transfer

Here /backup is the location where the files are downloaded.

Thanks for dropping by. Ready for the next blog?

cPanel to DirectAdmin Migration