Reading · menno.io · RSS Amplifier

This page cannot be shown here. You can still read it on the original site — the toolbar below keeps your place in the directory.

I recently had to write some NodeJS code which uses the AWS SDK to list all the objects in a S3 bucket which potentially contains many objects (currently over 80,000 in production). The S3 listObjects API will only return up to 1,000 keys at a time so you have to make multiple calls, setting the Marker field to page through all the keys. It turns out there's a lot of sub-optimal examples out…

I recently had to write some NodeJS code which uses the AWS SDK</a> to list all the objects in a S3</a> bucket which potentially contains many objects (currently over 80,000 in production). The S3 listObjects</a> API will only return up to 1,000 keys at a time so you have to make multiple calls, setting the Marker</code> field to page through all the keys.</p>

It turns out there's a lot of sub-optimal examples out there for how to do this which often involve global state and complicated recursive callbacks. I'm also a fan of the clarity of JavaScript's newer async/await feature for handling asynchronous code so I was keen on a solution which uses that style.</p>

Here's what I came up with:</p>

</span>
async function</span> allBucketKeys</span>(</span>s3</span>,</span> bucket</span>) {</span></span>
  const</span> params</span> =</span> {</span></span>
    Bucket: bucket,</span></span>
  };</span></span>
</span>
  var</span> keys</span> =</span> [];</span></span>
  for</span> (;;) {</span></span>
    var</span> data</span> = await</span> s3.</span>listObjects</span>(params).</span>promise</span>();</span></span>
</span>
    data.Contents.</span>forEach</span>((</span>elem</span>)</span> =></span> {</span></span>
      keys</span> =</span> keys.</span>concat</span>(elem.Key);</span></span>
    });</span></span>
</span>
    if</span> (</span>!</span>data.IsTruncated) {</span></span>
      break</span>;</span></span>
    }</span></span>
    params.Marker</span> =</span> data.NextMarker;</span></span>
  }</span></span>
</span>
  return</span> keys;</span></span>
}</span></span></code></pre>
It's called like this:</p>
</span>
// Remember to catch exceptions somewhere...</span></span>
const</span> s3</span> =</span> connectToS3Somehow</span>();</span></span>
var</span> keys</span> = await</span> allBucketKeys</span>(s3,</span> "my_bucket"</span>);</span></span></code></pre>
This solution is clean, concise and hopefully straightforward.</p>
An important aspect that supports this solution is that the AWS API
can return a Promise</a> for a call (via .promise()</code>) which can then be
used with await</code>. Given the need to conditionally call listObjects</code>
multiple times, an arguably clearer code structure can be achieved using
await instead of callbacks.</p>

Listing S3 objects with NodeJS

Comments