To download files from an Amazon S3 bucket recursively, you can use the AWS Command Line Interface (CLI) with the aws s3 cp command. The --recursive flag allows you to download all files and objects within a specified S3 directory and its subdirectories. Here's how to do it:
Using the AWS CLI to Download from an S3 Bucket Recursively:
1. Open a terminal or command prompt on your local machine.
2. Make sure you have the AWS CLI installed and configured with your AWS credentials. If you haven't configured it yet, you can run 'aws configure' to set up your access key, secret key, default region, and default output format.
[root@siddhesh ~]# aws configure
AWS Access Key ID : ********************************
AWS Secret Access Key : **********************************
[root@siddhesh ~]# 3. List the current data of the S3 bucket.
[root@siddhesh ~]# aws s3 ls mybucket-siddhesh-test --recursive
2023-10-30 18:17:04 48668 linux.jpeg
2023-10-30 18:17:19 0 rawlogs/
2023-10-30 18:19:00 0 rawlogs/app1_logs/
2023-10-30 18:19:29 0 rawlogs/app1_logs/app1.log
2023-10-30 18:19:29 0 rawlogs/app1_logs/app1.log.1
2023-10-30 18:19:29 0 rawlogs/app1_logs/app1.log.2
2023-10-30 18:18:27 0 rawlogs/maillog.1
2023-10-30 18:18:27 0 rawlogs/maillog.2
[root@siddhesh ~]#This will list all the objects in the 'mybucket-siddhesh-test' S3 bucket, including objects in any subdirectories within the bucket. This is useful when you want to list all objects within a bucket, especially if you have a hierarchical structure with multiple folders and subfolders in your S3 bucket.
4. Use the following command to download files recursively from an S3 bucket.
[root@siddhesh ~]# aws s3 cp s3://mybucket-siddhesh-test/ /root/data_download/ --recursive
download: s3://mybucket-siddhesh-test/linux.jpeg to data_download/linux.jpeg
download: s3://mybucket-siddhesh-test/rawlogs/app1_logs/app1.log to data_download/rawlogs/app1_logs/app1.log
download: s3://mybucket-siddhesh-test/rawlogs/app1_logs/app1.log.2 to data_download/rawlogs/app1_logs/app1.log.2
download: s3://mybucket-siddhesh-test/rawlogs/app1_logs/app1.log.1 to data_download/rawlogs/app1_logs/app1.log.1
download: s3://mybucket-siddhesh-test/rawlogs/maillog.1 to data_download/rawlogs/maillog.1
download: s3://mybucket-siddhesh-test/rawlogs/maillog.2 to data_download/rawlogs/maillog.2
[root@siddhesh ~]#where, mybucket-siddhesh-test : The name of the S3 bucket.
/root/data_download/ : The local directory where you want to save the downloaded files.
Verify downloaded data under a local directory. You will have the exact same directory as S3 on the local server.
[root@siddhesh ~]# ls -R /root/data_download/
/root/data_download/:
linux.jpeg rawlogs
/root/data_download/rawlogs:
app1_logs maillog.1 maillog.2
/root/data_download/rawlogs/app1_logs:
app1.log app1.log.1 app1.log.2
[root@siddhesh ~]#
Commenti