How to download a large AWS S3 bucket

Working around the limitations of S3

Introduction

Every year around the December holidays I embark on a particular backup routine. The emphasis is on backing up a large S3 bucket which contains the media files of JungleDragon, the wildlife community I founded.

The bucket currently is 612GB in size containing ~900K files.

The chance of S3 objects getting lost is almost infinitely small but the risk is not zero. Another risk may be account related: getting locked out of your AWS account or it being hacked.

A yearly full back-up is a compromise to not have an existential dependency on an external party. In the very unlikely case of a total data loss at AWS side, I’m able to recover almost all data. Losing on average 6 months of data is preferable over losing it all, which would be a game over event. Running this full backup routine more often would make sense but the problem is that it is costly. Getting things out of S3 is very expensive.

Anyway, for multiple years in a row I’ve faced huge challenges doing this full backup, so this year I finally got to the bottom of it and found a solution. By documenting it, hopefully it helps others with similar problems.

The problem

The AWS S3 console has no option to download a bucket. Instead, it’s recommended to use the AWS CLI tool. The most straightforward way to download a bucket is the cp command:

aws s3 cp s3://yourbucketaddress . — recursive

When I run this on my large bucket it starts downloading at about 20Mbit/s on a 400Mbit/s connection. So it’s very slow. But it gets worse. The longer it runs, the more it slows down. About an hour into the process, it may be at 5Mbit/s. An hour later at 2Mbit/s. And so on. As a result, the total completion time grows exponentially.

As the total runtime grows into days, errors inevitably occur. AWS may simply error out on a specific file or the process is killed altogether. At this point you’d have a partial backup which might even contains gaps. You can’t easily resume downloading. A way around this problem is to segment your large bucket into multiple commands, but overall the cp command should be considered unsafe.

The better command is sync:

aws s3 sync s3://yourbucketaddress . — recursive

This command can be restarted. In the case of a fatal error, you just run the same command and it will resume downloading objects not yet downloaded. In order to do this, it has to compare local files to your online objects, which takes forever. The second time you run this command on a large bucket, it may take a full hour before downloading actually resumes.

Further, this command has the exact same slowdown problem as the cp command. It is slow, gets ever slower the longer it runs, and this way it still takes days to make a backup.

It’s hard to say how universal these slowdown problems are, but I’ve found many people reporting the same issue. The CLI uses a fraction of available bandwidth, is unable to maintain pace and is generally unsafe.

I find it astonishing that this pinnacle of cloud services is so very bad at this basic use case whilst at the same time charging a fortune for it.

The solution

The solution is quite simple once you know it:

GitHub - peak/s5cmd: Parallel S3 and local filesystem execution tool.

Through an obscure comment on StackOverflow I came across this tool making bold claims about high performance S3 operations:

Too good to be true? Nope!

I’m on Windows, so let’s first cover installation:

• Be sure that you have a fully functioning AWS CLI setup. It seems that s5cmd reuses the credentials (and perhaps other parts)
• Download the latest Windows release (click on “show all assets” in case you don’t see it)
• Extract the zip towards a stable path
• Optional: add the path to your Windows environment variables so that you can run the command from any folder

An example command:

s5cmd sync s3://yourbucketaddress/* .

This will recursively download (sync) the entire bucket to your current directory.

On my first sustained test of 33.4GB (42,000 files) I got a transfer+write rate of 266Mbit/s on a 400Mbit/s connection. Which is 12.6 times(!) faster compared to the AWS CLI tool.

Further, I found no meaningful slowdown taking place on the full backup. It overall ranged between 400Mbit/s and 200Mbit/s sustained, without a general downward trajectory.

The disk problem

My next bottleneck was write speed. I did not have enough free disk space to download the entire bucket to my PC’s main SSD so I instead ran the command on an external USB drive, a 5 year old 2TB Transcend disc. This disc is specified as having a maximum write speed of 5Gb/s but it couldn’t even keep up with a fraction of that. All the gains from using s5cmd were lost.

So I invested in a new disk, a Samsung T9. Using manual testing (just dragging files to it), it writes at 1000MB/s. Which is ridiculously fast but still only half of what it can do. The limitation is in me not having a USB-C 3.2 port on my PC.

Needless to say, it could easily keep up with writing ~250Mbit/s in combination with the s5cmd tool.

And that’s how my frustrating annual backup routine is now silky smooth and fast. Concluding:

• Amazon, be very ashamed.
• Makers of s5cmd: you’re heroes.

Happy holidays!

How to download a large AWS S3 bucket was originally published in Ferdy Christant on Medium, where people are continuing the conversation by highlighting and responding to this story.