S3: List objects using filters

One thing that really sucks about the Amazon S3 command is not being able to efficiently list objects using the exclude and include filters.

I recently had a case when I needed to list GZip archives from one of our S3 buckets but only the ones ending by .gzip.

Since the ls command doesn't support filters (yet?), I found another way to get that listing: use the cp command, but in dry-run mode:

#!/bin/bash
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

# Very important: DO NOT remove the --dryrun flag!
aws s3 cp s3://bucket-name . --recursive --no-progress --dryrun \
--exclude '*' --include '*.gzip' \
| cut -d' ' -f3

With this script, you'll get a clear list of your object full paths ending with .gzip.

If you don't have the newest version of Amazon command available in your environment (no support of the --no-progress flag), you can use this version of the script:

#!/bin/bash
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

# Very important: DO NOT remove the --dryrun flag!
aws s3 cp s3://bucket-name . --recursive --dryrun \
--exclude '*' --include '*.gzip' \
| tr "\r" "\n" \
| grep -v '^Completed ' \
| cut -d' ' -f3

Thank you Julien for the hint.

Happy listing!