download s3 bucket - How to Download an Entire S3 Bucket or Folder with aws s3 sync and aws s3 cp --recursive (Real Output), Plus MinIO mc, s3cmd and Cyberduck

TL;DR

• The command: aws s3 sync s3://<bucket>[/prefix] /local/dir. It downloads everything under the bucket or prefix, keeps the folder structure, and on the next run transfers only what changed - the second sync in my test printed nothing and exited 0.
• Measure first: aws s3 ls s3://<bucket>/<prefix> --recursive --summarize --human-readable | tail -2 tells you how much you are about to pull. The prefix in this post is 113.5 GiB in 141,310 objects, which is why the examples filter it down to 5.6 MiB.
• Filter: --exclude '*' --include 'AE*.csv' in that order (the last matching rule wins). --dryrun prints what would be downloaded without downloading.
• aws s3 cp --recursive does the same first download, but re-downloads everything on every run. Use it for a one-off copy, sync for anything you will repeat.
• No credentials? Public buckets work with --no-sign-request; everything else needs an identity, and AWS CLI 2.37 now says so with Unable to locate credentials. You can configure credentials by running "aws login".
• The alternatives (MinIO mc, s3cmd, Cyberduck) are below for machines where the AWS CLI is not an option. Every CLI output in this post is from aws-cli 2.37.11, captured on 2026-10-12 on Windows.
• Not your exact case? The closest post on this site is Learn AWS S3.

Table of Content

1. Why the console cannot do it, and what the CLI does instead
2. Step 1 - List the bucket and measure what you are about to download
3. Step 2 - Download with aws s3 sync (with filters and a dry run)
4. sync versus cp --recursive
5. Errors you will see
6. Alternative 1 - MinIO mc
7. Alternative 2 - s3cmd
8. Alternative 3 - Cyberduck (GUI)
9. What broke when I ran this
10. Conclusion

1. Why the console cannot do it, and what the CLI does instead

Amazon S3 holds anything - text, images, data lakes, whole static websites - and the console lets you browse it, upload to it and download one object at a time. Select more than one and the Download button goes grey -

So "download the entire bucket" is a command-line job. The AWS CLI has two commands for it, and they behave differently -

If you have never set up the CLI, my AWS CLI install and setup guide covers it. Everything below works with any of the usual credential sources (aws configure, environment variables, an SSO profile, an instance role) and, for public buckets, with no credentials at all.

2. Step 1 - List the bucket and measure what you are about to download

I do not keep AWS credentials on the laptop I write on, so for this post I used a public bucket from the AWS Open Data registry - NOAA's daily climate observations, s3://noaa-ghcn-pds - with --no-sign-request, which tells the CLI not to sign the request. Drop that flag for your own buckets. Listing the top level -

 1$ aws s3 ls s3://noaa-ghcn-pds/ --no-sign-request
 2                           PRE csv.gz/
 3                           PRE csv/
 4                           PRE parquet/
 52023-02-04 00:32:30       3659 ghcnd-countries.txt
 62026-10-07 20:59:48   36395016 ghcnd-inventory.txt
 72023-02-04 00:32:29       1086 ghcnd-states.txt
 82026-10-07 20:59:47   11441354 ghcnd-stations.txt
 92026-10-12 03:00:55        270 ghcnd-version.txt
102025-11-24 22:33:53      36822 index.html
112025-04-01 18:20:10    4456595 mingle-list.txt
122023-01-20 15:44:31       1210 readme-by_station.txt
132023-02-04 00:32:30       1106 readme-by_year.txt
142026-10-06 01:59:14      29587 readme.txt
152023-01-20 15:44:22        173 status-by_station.txt
162023-01-20 16:40:22        169 status-by_year.txt
172026-10-01 00:57:59      38016 status.txt
182023-02-04 00:32:29          0 test.txt

PRE marks a prefix, which is what S3 calls a folder. Before downloading a prefix, find out how big it is - this is the step that saves you from filling a disk -

1$ aws s3 ls s3://noaa-ghcn-pds/csv/by_station/ --no-sign-request --recursive --summarize --human-readable | tail -2
2Total Objects: 141310
3   Total Size: 113.5 GiB

113.5 GiB in 141,310 files. At the 3.1 MiB/s my connection managed below that would be about 10 hours, which is exactly why the next section filters.

With your own account and no bucket name in mind, aws s3 ls with no arguments lists every bucket you own -

3. Step 2 - Download with aws s3 sync (with filters and a dry run)

The whole bucket into a local directory -

1aws s3 sync s3://jhooq-test-bucket-1 /home/vagrant/my-destination-directory

Two arguments: the source (s3://bucket or s3://bucket/prefix/) and the destination directory, which is created if it does not exist. For the post I restricted the 113.5 GiB prefix to the files starting with AE (the stations in the United Arab Emirates) -

 1$ aws s3 sync s3://noaa-ghcn-pds/csv/by_station/ /tmp/ghcn --no-sign-request --exclude '*' --include 'AE*.csv'
 2Completed 1.0 MiB/~5.6 MiB (977.0 KiB/s) with ~4 file(s) remaining (calculating...)
 3Completed 3.0 MiB/~5.6 MiB (2.0 MiB/s) with ~4 file(s) remaining (calculating...)
 4download: s3://noaa-ghcn-pds/csv/by_station/AEM00041218.csv to ghcn\AEM00041218.csv
 5download: s3://noaa-ghcn-pds/csv/by_station/AE000041196.csv to ghcn\AE000041196.csv
 6download: s3://noaa-ghcn-pds/csv/by_station/AEM00041217.csv to ghcn\AEM00041217.csv
 7download: s3://noaa-ghcn-pds/csv/by_station/AEM00041194.csv to ghcn\AEM00041194.csv
 8Completed 5.6 MiB/~5.6 MiB (3.1 MiB/s) with ~0 file(s) remaining (calculating...)
 9
10$ ls -la /tmp/ghcn | head -8
11total 6280
12-rw-r--r-- 1 HP 197121 1998959 Oct 12 03:17 AE000041196.csv
13-rw-r--r-- 1 HP 197121 1597557 Oct 12 03:17 AEM00041194.csv
14-rw-r--r-- 1 HP 197121 1382749 Oct 12 03:17 AEM00041217.csv
15
16$ du -sh /tmp/ghcn
175.7M    /tmp/ghcn

(The progress lines overwrite each other in a terminal; I kept three of them, and my head -8 cut the fourth file, AEM00041218.csv, from the listing - du shows all four.) Four files, 5.6 MiB, at 3.1 MiB/s. Three things in that command worth knowing -

• --exclude '*' --include 'AE*.csv' - filters are applied in order and the last match wins, so you exclude everything first and then include what you want. Quote the patterns or the shell expands * against your local files.
• The local timestamps (Oct 12 03:17) are S3's last-modified times, not the download time; that is how sync knows what changed next time.
• --no-sign-request is only for public buckets. With your own bucket, leave it out and the CLI uses your configured credentials.

Run the same command again and nothing happens - that is the whole point of sync -

1$ aws s3 sync s3://noaa-ghcn-pds/csv/by_station/ /tmp/ghcn --no-sign-request --exclude '*' --include 'AE*.csv'

(Empty output, exit code 0: every local file matched its S3 object's size and timestamp.)

Before a big download, see what would happen with --dryrun -

1$ aws s3 sync s3://noaa-ghcn-pds/csv/by_station/ /tmp/ghcn-dry --no-sign-request --exclude '*' --include 'AF*.csv' --dryrun
2(dryrun) download: s3://noaa-ghcn-pds/csv/by_station/AF000040930.csv to ghcn-dry\AF000040930.csv
3(dryrun) download: s3://noaa-ghcn-pds/csv/by_station/AFM00040938.csv to ghcn-dry\AFM00040938.csv
4(dryrun) download: s3://noaa-ghcn-pds/csv/by_station/AFM00040948.csv to ghcn-dry\AFM00040948.csv
5(dryrun) download: s3://noaa-ghcn-pds/csv/by_station/AFM00040990.csv to ghcn-dry\AFM00040990.csv

Other flags I use with sync -

The full list is in the aws s3 sync reference, and the general S3 command patterns in the CLI S3 commands guide.

4. sync versus cp --recursive

aws s3 cp with --recursive downloads the same files -

1$ aws s3 cp s3://noaa-ghcn-pds/csv/by_station/ /tmp/ghcn-cp --recursive --no-sign-request --exclude '*' --include 'AE*.csv'
2download: s3://noaa-ghcn-pds/csv/by_station/AE000041196.csv to ghcn-cp\AE000041196.csv
3download: s3://noaa-ghcn-pds/csv/by_station/AEM00041218.csv to ghcn-cp\AEM00041218.csv
4download: s3://noaa-ghcn-pds/csv/by_station/AEM00041194.csv to ghcn-cp\AEM00041194.csv
5download: s3://noaa-ghcn-pds/csv/by_station/AEM00041217.csv to ghcn-cp\AEM00041217.csv
6Completed 5.6 MiB/~5.6 MiB (2.8 MiB/s) with ~0 file(s) remaining (calculating...)

but it has no memory: run it again and all 5.6 MiB come down again. For a bucket you download once, cp --recursive is fine and marginally simpler. For a bucket you will download again - a backup, a dataset you refresh, a site you mirror - sync is the only sensible choice, and sync --delete is how you keep an exact copy.

5. Errors you will see

1$ aws s3 ls s3://noaa-ghcn-pds/
2aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials. You can configure credentials by running "aws login".
3
4$ aws s3 ls s3://this-bucket-does-not-exist-jhooq-12345/ --no-sign-request
5aws: [ERROR]: An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

6. Alternative 1 - MinIO mc

MinIO is an S3-compatible object store, and its client mc talks to any S3 endpoint, AWS included. It is useful when you work with several S3-compatible systems and want one tool, and it shows per-file progress bars. Installation on Linux is one binary -

1wget https://dl.min.io/client/mc/release/linux-amd64/mc
2chmod +x mc
3sudo mv mc /usr/local/bin/

Then create an alias for AWS with an access key and secret key from IAM (never the root account keys) -

1mc alias set s3 https://s3.amazonaws.com <ACCESS_KEY> <SECRET_KEY>
2mc alias ls

and download the bucket recursively -

1mc cp --recursive s3/jhooq-test-bucket-1 .
2# ...cket-1/test.txt:  146.56 KiB / 146.56 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 406.00 KiB/s 0s

mc mirror s3/jhooq-test-bucket-1 ./local is the sync equivalent (only changed objects, with --remove for an exact mirror). The command reference is the mc documentation.

7. Alternative 2 - s3cmd

s3cmd is the oldest of the three and still popular on servers where it is already installed. Today it is a pip package (the 2021 version of this post unzipped the source; do not bother) -

1python3 -m pip install s3cmd
2s3cmd --configure     # asks for the access key, secret key, region; writes ~/.s3cfg

The configure dialog ends with a connection test - Success. Your access key and secret key worked fine :-) - and then the download is -

1s3cmd get --recursive s3://jhooq-test-bucket-1 /home/vagrant/s3-bucket/

1download: 's3://jhooq-test-bucket-1/AGL Payment Receipt.pdf' -> '/home/vagrant/s3-bucket/AGL Payment Receipt.pdf'  [1 of 3]
2 52562 of 52562   100% in    0s   505.08 KB/s  done
3download: 's3://jhooq-test-bucket-1/Screenshot 2021-10-28 at 23.45.36.png' -> '/home/vagrant/s3-bucket/Screenshot 2021-10-28 at 23.45.36.png'  [2 of 3]
4 97511 of 97511   100% in    0s   448.68 KB/s  done
5download: 's3://jhooq-test-bucket-1/test.txt' -> '/home/vagrant/s3-bucket/test.txt'  [3 of 3]

s3cmd sync s3://bucket /local/ is its incremental mode. One warning: s3cmd stores the secret key in plain text in ~/.s3cfg; keep the file at chmod 600.

8. Alternative 3 - Cyberduck (GUI)

If you would rather not type, Cyberduck (Windows and macOS, free) opens an S3 bucket like a file browser: add a connection of type Amazon S3 with your access key and secret key, browse, select everything, download. It also handles the multi-select download that the AWS console refuses -

For anything scripted, scheduled or larger than a few GB, use the CLI; Cyberduck is for the occasional manual pull.

9. What broke when I ran this

I re-ran the CLI part of this post on 2026-10-12 with aws-cli 2.37.11 on Windows, against the public NOAA bucket, because I do not keep AWS credentials on this laptop. Four things changed or surprised me.

1. The no-credentials error now points at aws login. The 2021 CLI said Unable to locate credentials. You can configure credentials by running "aws configure". Version 2.37 says by running "aws login", the newer SSO-style sign-in. aws configure with an access key still works; the hint just changed. And note the CLI refuses even a public bucket without credentials unless you pass --no-sign-request - it does not fall back to anonymous access on its own.

2. A non-existent prefix is not an error. aws s3 sync s3://noaa-ghcn-pds/no/such/prefix/ /tmp/nothing printed nothing and exited 0. In a script that means a typo in the prefix looks like "nothing to do". Check the exit code and that the destination has files, or run aws s3 ls on the prefix first.

3. The progress output is one very long line. The CLI redraws Completed 1.0 MiB/~5.6 MiB ... with carriage returns, so in a log file it becomes a single 2,000-character line. --only-show-errors or --no-progress keeps logs readable; I trimmed the lines in this post by hand.

4. Measuring first was the right call. The prefix I picked for the demo turned out to be 113.5 GiB. At the 3.1 MiB/s I was getting that is roughly 10 hours and a full disk. --summarize took 79 seconds to walk the 141,310 objects (listing is paged 1,000 keys at a time) and saved me from that; put it in front of every bulk download you have not done before.

10. Conclusion

To download an entire S3 bucket, aws s3 sync s3://bucket /local/dir is the answer for almost everyone: it needs no extra tools, it resumes, and run again it transfers only what changed. Measure the bucket with --summarize first, filter with --exclude and --include, and preview with --dryrun. cp --recursive is the same download without the memory; mc, s3cmd and Cyberduck are there for environments where the AWS CLI is not. For the scripting side, wrap the command in an Ansible playbook or a cron job with --only-show-errors, and if the bucket is one you create yourself, my S3 course covers buckets, policies and the CLI from the start.