One thing that really sucks about the Amazon S3 command is not being able to efficiently list objects using the exclude and include filters.
I recently had a case when I needed to list GZip archives from one of our S3 buckets but only the ones ending by .gzip.
Since the ls command doesn't support filters (yet?), I found another way to get that listing: use the cp command, but in dry-run mode:
#!/bin/bash
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
# Very important: DO NOT remove the --dryrun flag!
aws s3 cp s3://bucket-name . --recursive --no-progress --dryrun \
--exclude '*' --include '*.gzip' \
| cut -d' ' -f3
With this script, you'll get a clear list of your object full paths ending with .gzip.
If you don't have the newest version of Amazon command available in your environment (no support of the --no-progress flag), you can use this version of the script:
#!/bin/bash
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
# Very important: DO NOT remove the --dryrun flag!
aws s3 cp s3://bucket-name . --recursive --dryrun \
--exclude '*' --include '*.gzip' \
| tr "\r" "\n" \
| grep -v '^Completed ' \
| cut -d' ' -f3
Thank you Julien for the hint.
Happy listing!