The Amazon Simple Queue Service (SQS) is a fully managed message queue service that enables decoupling and communication between microservices, distributed systems, and serverless applications. SQS facilitates asynchronous and reliable message exchange between different components of a system, helping to build highly scalable and resilient solutions. It is important to emphasize making our solution more resilient and highlighting the asynchronous nature of communication.
Key Features of SQS
Reduces Coupling Between Integrations: Allows different parts of an application to communicate without relying on each other’s immediate availability.
Automatic Scalability: Capable of handling high message volumes, automatically scaling to meet potential load variations without the need to manage the infrastructure.
Security: Provides granular access control with AWS Identity and Access Management (IAM) and encryption in transit (TLS) and at rest (AWS KMS).
Queue Types: Supports Standard and FIFO (First-In-First-Out) queues.
Integration with AWS Services: Easily integrates with other AWS services such as AWS Lambda, Amazon S3, Amazon EC2, and Amazon SNS.
Fundamental Concepts
a) Queues
A queue is a temporary repository for messages awaiting processing. SQS offers two types of queues:
- Standard Queues: Provide unlimited throughput, at-least-once delivery, and best-effort message ordering.
Here, it is worth opening a quick note to discuss two key features of standard queues: (i) at-least-once delivery and (ii) best-effort message ordering.
(i) At-Least-Once Delivery (Documentation): This characteristic highlights the need to handle potential message duplication when using standard SQS queues. Here, a key concept is idempotency.
(ii) Best-Effort Message Ordering: Standard queues do not guarantee exact ordering. To ensure processing in the same order as sending, use FIFO (First-In-First-Out) queues.
- FIFO Queues: Guarantee that messages are processed exactly once and in the order they were sent. Ideal for applications requiring ordered processing.
b) Messages
A message is the unit of data sent to the queue (in other words, what is sent). Messages can contain up to 256 KiB of plain text or JSON data. Messages remain in the queue until consumed or expired. For message limits, refer to Message Quotas.
c) Consumers
Consumers are applications or services that receive and process messages from the queue. They can run on any platform or in any language that supports SQS API calls.
d) Message Visibility
Visibility Timeout is the period after a consumer retrieves a message during which that message remains invisible to other consumers. If processing fails, the message becomes visible again after this period. This mechanism adds resilience, as longer retry intervals may be beneficial in some situations.
Getting Started with Amazon SQS
Prerequisites
- Access to the AWS Management Console.
Step 1: Create a Queue
Using the AWS Console
Access the AWS Management Console.
Search for SQS.
On the Amazon SQS page, click Create queue.
Under Queue name, enter a name for your queue, e.g., MyTestQueue.
Choose the queue type:
Standard: For high throughput and best-effort message ordering.
FIFO: For strict ordering and exactly-once processing. FIFO queue names must end with
.fifo.
For this example, select Standard.
Optionally, configure Default or Advanced queue settings, such as visibility timeout, message retention time, maximum message size, and access policies.
Click Create queue.
Step 2: Send Messages to the Queue
In the queue list, click on the name of the queue you created (MyTestQueue).
Click the Send and receive messages tab.
Under Send a message, enter the message content in the Message field.
- You can enter plain text or JSON.
Optionally, configure message attributes or delivery delay.
Click Send message.
→ In this tutorial, we will continue developing the text of our Lambda function. The text can be seen at the following link: https://marceloamorim.hashnode.dev/introduction-to-aws-lambda.
- Initially, let’s modify the existing code in the Lambda function to:
import json
import os
import boto3
from botocore.exceptions import ClientError
# Initialize SQS client
sqs_client = boto3.client('sqs')
def lambda_handler(event, context):
# Retrieve the queue URL from environment variables
queue_url = os.getenv('SQS_QUEUE_URL')
# Hardcoded message
message = "Testing lambda function integration with SQS"
# Log the event for debugging
print(f"Event received: {json.dumps(event)}")
print(f"Sending message to queue: {queue_url}")
try:
# Send the message to the SQS queue
response = sqs_client.send_message(
QueueUrl=queue_url,
MessageBody=message
)
print(f"Message sent successfully: {response['MessageId']}")
return {
'statusCode': 200,
'body': json.dumps({
'message': message,
'sqsMessageId': response['MessageId']
})
}
except ClientError as e:
print(f"Error sending message to SQS: {e}")
return {
'statusCode': 500,
'body': json.dumps({
'error': str(e)
})
}
Remember to add the environment variable SQS_QUEUE_URL with the URL of your queue.
To ensure everything works properly, you must associate an IAM Role with this function that allows publishing messages to the queue:
→ Finally, we will receive this result, which indicates that the message is in our queue:
Step 3: Receive and Delete Messages
→ After the message is published to the queue, we have a message available:
Still in the Send and receive messages tab, scroll to the Receive messages section.
Click Start polling to make the console begin polling for messages in the queue.
The available messages will be listed below. You can view the content by clicking on the message.
To simulate processing, select a message and click Delete to remove it from the queue after processing.
Confirm the deletion when prompted.
Step 4: Configure a Consumer
For real applications, you would configure a consumer to process messages programmatically.
Example Using AWS Lambda
Navigate to the AWS Lambda service in the console.
Create a new Lambda function in the language of your choice (e.g., Python).
In the Designer, click Add trigger.
Select SQS as the trigger source.
Select the MyTestQueue queue.
Click Add.
In the Lambda function, write the code to process the received messages.
Example code in Python:
import json
def lambda_handler(event, context):
for record in event['Records']:
message = record['body']
print(f"Processing message: {message}")
return 'Success'
→ Now we have our function:
→ Which will receive messages in this format:
→ Searching the logs in CloudWatch, we verify that the message was successfully received:
→ Recap: where are we?
Initially, we created a Lambda function that publishes a message to our queue, and this message is received by another Lambda function.
→ Where are we going?
We will add an SNS topic before the queue and work with the fan-out concept:
More Concepts
a) Advanced Queue Configurations
Message Retention Time: The period that a message remains in the queue if not processed (1 minute to 14 days).
Visibility Timeout: The time a message remains invisible after being retrieved by a consumer (0 to 12 hours).
Delayed Message Delivery: Allows delaying the availability of new messages in the queue (0 to 15 minutes).
b) Dead-Letter Queues (DLQ)
- Dead-Letter Queues: These are queues used to store messages that could not be successfully processed after a defined number of attempts. This helps isolate and analyze problematic messages.
c) Access Control and Security
Use IAM policies to control who can interact with your queues.
Enable encryption at rest by integrating SQS with AWS KMS to protect the content of messages. It’s worth noting that KMS itself does not encrypt or decrypt the messages but manages the encryption keys. For more information, see https://docs.aws.amazon.com/en_us/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-server-side-encryption.html.
Final Considerations
Amazon SQS is a powerful tool for building distributed and scalable systems. Understanding the concepts of queues, messages, consumers, and the architectural contexts where SQS fits is essential to leveraging its capabilities. Using the AWS Management Console, you can quickly set up queues and start developing more resilient and decoupled applications. I hope this text has been helpful. Until next time! :)