timothymugayi/boto3-sqs-extended-client-lib

Boto3 SQS Extended Client Library for Python. An extension to the boto3 sqs client that enables sending and receiving messages up to 2GB via Amazon S3. [WARNING: This library is still under development contributors welcome]

Boto3 SQS Extended Client Library for Python

The Amazon SQS Extended Client Library for Python has been modelled after the original Amazon SQS Extended client library This python library enables you to manage Amazon SQS message payloads with Amazon S3. This is especially useful for storing and retrieving messages with a message payload size greater than the current SQS limit of 256 KB, up to a maximum of 2 GB. Specifically, you can use this library to:

• Specify whether message payloads are always stored in Amazon S3 or only when a message's size exceeds 256 KB.

• Send a message that references a single message object stored in an Amazon S3 bucket.

• Get the corresponding message object from an Amazon S3 bucket.

• Delete the corresponding message object from an Amazon S3 bucket.

Getting Started

• Sign up for AWS -- Before you begin, you need an AWS account. For more information about creating an AWS account and retrieving your AWS credentials, see [AWS Account and Credentials.

• Sign up for Amazon SQS -- Go to the Amazon SQS console to sign up for the service.

• Minimum requirements -- To use the sample application, you'll need python 2.7 recommended to use 3 (and above)

• Further information - Read the API documentation.

Installation

Install using pip...

pip install pysqs-extended-client

Usage

Standard Queues

    import base64

    from pysqs_extended_client.SQSClientExtended import SQSClientExtended
    
    # from string import ascii_letters, digits
    # from random import choice
    
    from pysqs_extended_client.config import (AWS_SQS_QUEUE_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION)
    
    if __name__ == '__main__':
    
        sqs = SQSClientExtended(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, 'tiptapcode-sqs-data')
    
        # _100mb_large_string = ''.join([choice(ascii_letters + digits) for i in range(104857600)])
    
        # message = "_100mb_large_string"
    
        message = None
        with open("C:\\DjangoCourse\\Courses\\celery\\introduction-promo.mp4", "rb") as image_file:
            encoded_string = base64.b64encode(image_file.read())
            message = encoded_string.decode("utf-8")
    
        sqs.send_message(AWS_SQS_QUEUE_URL, message)
    
        res = sqs.receive_message(AWS_SQS_QUEUE_URL)
    
        for message in res:
            sqs.delete_message(AWS_SQS_QUEUE_URL, message.get('ReceiptHandle'))

FIFO queues

    import base64
    import uuid

    from pysqs_extended_client.SQSClientExtended import SQSClientExtended
    
    # from string import ascii_letters, digits
    # from random import choice
    
    from pysqs_extended_client.config import (AWS_SQS_QUEUE_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION)
    
    if __name__ == '__main__':
    
        sqs = SQSClientExtended(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, 'tiptapcode-sqs-data')
    
        # _100mb_large_string = ''.join([choice(ascii_letters + digits) for i in range(104857600)])
    
        # message = "_100mb_large_string"
    
        message = None
        with open("C:\\DjangoCourse\\Courses\\celery\\introduction-promo.mp4", "rb") as image_file:
            encoded_string = base64.b64encode(image_file.read())
            message = encoded_string.decode("utf-8")

        # Create group and deduplication IDs, required for FIFO queues.
    		message_group_id = str(uuid.uuid4())
        message_deduplication_id = str(uuid.uuid4())

        sqs.send_message(AWS_SQS_QUEUE_URL, message, message_group_id=message_group_id, message_deduplication_id=message_deduplication_id)
    
        res = sqs.receive_message(AWS_SQS_QUEUE_URL)
    
        for message in res:
            sqs.delete_message(AWS_SQS_QUEUE_URL, message.get('ReceiptHandle'))

Feedback

We use GitHub issues for tracking bugs and feature requests and have limited bandwidth to address them. Please use these community resources for getting help:

• Give feedback here.
• If you'd like to contribute a new feature or bug fix, go ahead submit a pull request.

boto3-sqs-extended-client-lib's Issues

sending more than one sqs message

Expected Behavior

I would like to send multiple sqs messages

Current Behavior

I am able to send one sqs message but not more than that.

Failure Information (for bugs)

RESERVED_ATTRIBUTE_NAME.value

sqs.send_message(AWS_SQS_QUEUE_URL,message[0])
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.6/dist-packages/pysqs_extended_client/SQSClientExtended.py", line 208, in send_message
raise ValueError("Message attribute name {} is reserved for use by SQS extended client.".format(SQSExtendedClientConstants.RESERVED_ATTRIBUTE_NAME.value))
ValueError: Message attribute name SQSLargePayloadSize is reserved for use by SQS extended client.

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

1. from pysqs_extended_client.SQSClientExtended import SQSClientExtended
2. sqs = SQSClientExtended(None,None,None, loc)
3. sqs.send_message(AWS_SQS_QUEUE_URL,message[0])

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

• Firmware Version:
• Operating System:
• package version:

Failure Logs

Please include any relevant log snippets or files here.

Message attribute name SQSLargePayloadSize is reserved for use by SQS extended client.

For some reason, the following three lines cause the client to crash with the message Message attribute name SQSLargePayloadSize is reserved for use by SQS extended client. sometimes:

boto3-sqs-extended-client-lib/sqs_client/SQSClientExtended.py

Lines 205 to 207 in f9a6cff

	large_payload_attribute_value = message_attributes.get(SQSExtendedClientConstants.RESERVED_ATTRIBUTE_NAME.value)
	if large_payload_attribute_value:
	raise ValueError("Message attribute name {} is reserved for use by SQS extended client.".format(SQSExtendedClientConstants.RESERVED_ATTRIBUTE_NAME.value))

Workaround: when removing those lines, I see no more error and everything works as expected every time.

The way I call the send_message method is using the following signature without adding any message attribute:
sent_message = sqs.send_message(url, message)

ie. message_attributes should be equal to {} and should not contain the so-called RESERVED_ATTRIBUTE_NAME right? I did not investigate why the above ValueError would be raised from time to time. Just leaving an issue here if anyone wants to go further or encounters the same problem as the workaround is pretty simply to remove this check and exception raise.

ACL not configurable

Expected Behavior

Functionality to be able to read a message from a different account to the one that publishes the message.

Current Behavior

ACL is always set to "private" when uploading the file to S3, so it is not possible to read the message from a different account.
As a work-around I tried assuming a role in the other account, but it seems the credentials passed to the SQS client are ignored, in favor of the environment variables set in the config.py file.

Incompatible message format compared to amazon-sqs-java-extended-client-lib

The message format produced by "boto3-sqs-extended-client-lib" and "amazon-sqs-java-extended-client-lib" are different if the message size is greater than 256 KB.

boto3-sqs-extended-client-lib (JSON object)

e.g.
{"s3BucketName":"some-bucket","s3Key":"some-uuid"}

amazon-sqs-java-extended-client-lib (JSON array)

e.g.
["com.amazon.sqs.javamessaging.MessageS3Pointer",{"s3BucketName":"some-bucket","s3Key":"some-uuid"}]

The purpose of raising this issue is to align the message format with Amazon SQS Extended Client Library for Java mentioned in the official documentation.

References:
https://docs.amazonaws.cn/en_us/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html
https://github.com/awslabs/amazon-sqs-java-extended-client-lib

Add Support for FIFO Queues

Expected Behavior

I want send_message to accept message group ID and deduplication ID as a parameters so that FIFO queues can be pushed to from this utility.

Current Behavior

Currently the client has no support for message group and deduplication IDs, so it only supports standard queues.

What is the current behavior?

Failure Information (for bugs)

Pushing to a FIFO queue from this client raises the following botocore error:

botocore.exceptions.ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

1. Create a .fifo queue
2. Attempt to push to the queue via SQSClientExtended.send_message

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

• Firmware Version: 1.14.0
• Operating System: Ubuntu 19.04
• package version: Latest

Dependency resolving problem by Poetry

Expected Behavior

Poetry should be able to install this lib.

Current Behavior

but it gives error:

Because moto (1.3.13) depends on botocore (>=1.12.86)                                                                                                                                                           
   and pysqs-extended-client (0.0.1) depends on botocore (1.10.80), moto (1.3.13) is incompatible with pysqs-extended-client (0.0.1).                                                                             
  And because pysqs-extended-client (0.0.1) depends on moto (1.3.13), pysqs-extended-client is forbidden. 

Failure Information (for bugs)

see above

Steps to Reproduce

1. install poetry
2. add pysqs-extended-client = "*" to [tool.poetry.dependencies]
3. run poetry install

Context

python 3.8
mac intel chip

My guess

pip install can install it successfully probably it ignore dependency moto which is not used actually?
If moto is not really used, removing it can solve the problem.

PyPi is not up to date

Expected Behavior

PyPi is up to date with Github code.

Current Behavior

PyPi code is 3 years out of date.

Steps to Reproduce

1. Run pip install pysqs-extended-client
2. Check files against the github files
3. The files do not include the FIFO code.