zzsi/s3watcher

s3watcher's Issues

S3Watcher

🚀 Feature Request

Implement an S3Watcher class to watch S3 events using an existing SQS queue.

🔈 Motivation

This is the main functionality of this repo.

📎 Additional context

Reference: https://github.com/dbnicholson/s3watcher.

Set up SQS and S3 event notification

🚀 Feature Request

S3Watcher.watch() should trigger the provisioning of a SQS queue that listens to S3 event notifications. boto3 can be used to handle the infrastructure setup.

🔈 Motivation

This is necessary for S3Watcher to work.

🛰 Alternatives

Terraform and pulumi can be good declarative tools if we don't use boto3.

📎 Additional context

The s3 event notification configuration may look like (in XML, though JSON can be used too):

<NotificationConfiguration>
  <QueueConfiguration>
      <Id>1</Id>
      <Filter>
          <S3Key>
              <FilterRule>
                  <Name>prefix</Name>
                  <Value>images/</Value>
              </FilterRule>
              <FilterRule>
                  <Name>suffix</Name>
                  <Value>jpg</Value>
              </FilterRule>
          </S3Key>
     </Filter>
     <Queue>arn:aws:sqs:us-west-2:444455556666:s3notificationqueue</Queue>
     <Event>s3:ObjectCreated:Put</Event>
  </QueueConfiguration>
</NotificationConfiguration>

Reference:

• https://gist.github.com/reedobrien/c1af2a171b9bc5317606
• https://aws.amazon.com/blogs/storage/reliable-event-processing-with-amazon-s3-event-notifications/