Issue a signed URL with AWS SQS

AWS S3 has a "signed URL". SQS can do something similar to this feature.

When do you use it?

For example, suppose you want to create a system like this.

device_image.png

Consider the configuration on AWS.

** <Configuration A: API Gateway + Lambda> **

Lambda + API Gateway has a reputation for being cheap, but it becomes expensive when handling a large number of requests.

talk_image_gateway.png

It's a rough calculation, but when there are 172.8 million requests a day, the charge for accumulating data is as follows.

service Price per million requests Daily charge conditions
Lambda Depends on the conditions 387 USD Average 1 second processing, memory allocation 128MB
Api Gateway 4.25 USD 731 USD REST API
SQS 0.4 USD 69 USD

Total: $ 1187 per day

With Lambda delay, WAF, CloudFront, authentication, etc., it goes up even more.

** <Configuration B: AWS IoT + Rule Engine + SQS> **

If you have a configuration that transfers from AWS IoT to SQS with a rule engine, the price will go down. But this time ...

service Price per million requests Daily charge conditions
AWS IoT Send: 1.2 USD
Rule: 0.18 USD
Action: 0.18 USD
268 USD rule+Take action
SQS 0.4 USD 69 USD

Total: $ 337 per day

talk_image_iot.png

This time, MQTT is impossible due to the port, and WebSocket is also impossible due to the terminal side.

** **

By the way, if you can send directly from the terminal to SQS, it will cost about 1/20, $ 69 every day. There is a security problem because AWS-SDK cannot be used. Something more than that ...

talk_image_sqs.png

It's Sorya likely.

** <Configuration D: SQS + signed URL> **

If you think about it carefully, it seems that you can do it well if you can put authentication that does not use SDK and send it directly to SQS.

In order to create such a mechanism, we will realize SQS + signed URL.

Completion drawing

draw.png

** The processing to be executed and the execution timing of each are as follows **

processing Target timing
Get SQS signed URL Terminal → API Gateway Run once every 15 minutes
Send terminal status to SQS Terminal → SQS Run once every 5 seconds

** The data thrown by SQS is assumed to be as follows this time **

Data to send Data content
SQS queue name sqs-send-request-test-0424
Pattern of data to send Open/Open (terminal on, human feeling on)
Close/Open (terminal off, human feeling on)
Open/Close (terminal on, human feeling off)
Close/Close (terminal off, human feeling off)
* Send one of 4 ways

Get SQS signed URL (run once every 15 minutes)

Request a signed URL from API Gateway [^ 1]

[^ 1]: * API Gateway authentication is omitted this time. If you want to authenticate, please put the authentication information in the header of the transmitted data.

Terminal side request


curl "https://xxxxxxxxx.execute-api.ap-northeast-1.amazonaws.com/url" -s -X POST \
-d '{"que_name":"sqs-send-request-test-0424", "patterns":["Open/Open","Close/Open","Open/Close","Close/Close"]}'

If the request is successful, you will receive the same number of signed URLs as the data pattern.

Received data on the terminal side


{
    "url": {
        "Open/Open": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxx&Action=SendMessage&MessageBody=Open%2FOpen&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx", 
        "Close/Open": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxx&Action=SendMessage&MessageBody=Close%2FOpen&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx",
        "Open/Close": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxxx&Action=SendMessage&MessageBody=Open%2FClose&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx", 
        "Close/Close": "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxxx&Action=SendMessage&MessageBody=Close%2FClose&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx"
    }
}

Send terminal status to SQS (execute once every 5 seconds)

The URL to send is the data of data ["url"] [$ {state you want to send}]. In the example, "Open / Close" is sent.

Terminal side request


curl "https://sqs.ap-northeast-1.amazonaws.com/xxxxxx/sqs-send-request-test-0424?AWSAccessKeyId=xxxxxx&Action=SendMessage&MessageBody=Open%2FClose&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-04-30T10%3A42%3A54&Version=2012-11-05&Signature=xxxxxxxxxxxxxxxxxxxxx"

Every time you execute it, a $ {state you want to send} message is registered in the set topic. In the example, "Open / Close" is registered in "sqs-send-request-test-0424".

capture_2.png

Implementation

As a preliminary preparation, make a queue with SQS. Enter the name of the queue you like and click "Quick Queue Creation" at the bottom of the screen.

capture_5.png

Also, create an IAM user dedicated to SQS. Set AmazonSQSFullAccess to the policy. No permissions other than SQS are required.

capture_3.png

After creating the IAM user, get the access key ID and secret access key.

Implement Lambda

Create a Lambda with Python 3.8.

Set the Lambda environment variables as follows.

Environment variable key Value to set
AWS_ACCOUNT_NUMBER AWS account ID number
(The 12-digit number at the bottom left of the IAM screen)
SQS_ACCOUNT_ID IAM user access key ID dedicated to SQS
SQS_SECRET_KEY SQS-only IAM user secret access key

Paste the following source into Lambda.

Lambda



# coding: utf-8
import boto3, json, hashlib, hmac, base64, os 
from datetime import datetime, timezone, timedelta
from urllib import parse as url_encode

#Signature information (SHA256 format, signature version 2)
SIGNATURE_METHOD = "HmacSHA256"
SIGNATURE_VERSION = "2"
ENCODE = "utf-8"

#SQS method information (send message to Queue using REST API GET)
HTTP_METHOD = "GET"
SQS_METHOD = "SendMessage"
AWS_VERSION = "2012-11-05"

#Time zone
UTC_TIMEZONE = timezone(timedelta(hours = 0), 'UTC')

class Credentials:
    """
Set credentials from IAM user
    """
    @staticmethod
    def from_iam():
        instance = Credentials()
        instance.aws_access_key = os.environ.get("SQS_ACCOUNT_ID", DEFAULT.SQS_ACCOUNT_ID)
        instance.aws_secret_key = os.environ.get("SQS_SECRET_KEY", DEFAULT.SQS_SECRET_KEY)
        return instance

class Endpoint:
    """
Set SQS endpoint information
    """
    def __init__(self, topic_name):
        self.protocol = "https"
        self.host_name = "sqs.{}.amazonaws.com".format(os.environ.get("AWS_REGION", DEFAULT.AWS_REGION))
        self.url_path = "/{}/{}".format(os.environ.get("AWS_ACCOUNT_NUMBER", DEFAULT.AWS_ACCOUNT_NUMBER), topic_name)

    @property    
    def url(self):
        return f"{self.protocol}://{self.host_name}{self.url_path}"

#Create an SQS signed URL
def create_presigned_url(credential, endpoint, message):

    #Get the current date and time in UTC and convert it to a string
    current_date = datetime.now().astimezone(UTC_TIMEZONE)
    current_date_str = url_encode.quote(
        current_date.strftime('%Y-%m-%dT%H:%M:%S')
    )

    #Create a hash with a secret access key based on the data to be sent
    return endpoint.url + "?" + create_query(SQS_METHOD, message, credential.aws_access_key, current_date_str, option = {
        #Since the hash data is a byte array, URL encode + base64 encode so that it can be sent by GET.
        "Signature" : url_encode.quote(
            base64.b64encode(
                sign(
                    #Specify the secret access key to use for signing
                    credential.aws_secret_key.encode(ENCODE), 
                    #Specify the data to be sent (the specified data is hashed with SHA256)
                    create_certificate(endpoint, SQS_METHOD, message, credential.aws_access_key, current_date_str)
                )
            )
        )
    })

#Hash with secret access key
def sign(key, msg):
    return hmac.new(key, msg.encode(ENCODE), hashlib.sha256).digest()

#Create signature data in v2 format
#See official documentation for format (https)://docs.aws.amazon.com/ja_jp/general/latest/gr/signature-version-2.html)
def create_certificate(endpoint, method, message_body, aws_access_key, current_date_str):
    return "\n".join([
        HTTP_METHOD,
        endpoint.host_name,
        endpoint.url_path,
        create_query(method, message_body, aws_access_key, current_date_str)
    ])

#Normalize query data
#See official documentation for format (https)://docs.aws.amazon.com/ja_jp/general/latest/gr/signature-version-2.html)
def create_query(method, message_body, aws_access_key, current_date_str, option = None):
    query_map = {
        "AWSAccessKeyId" : aws_access_key,
        "Action" : method,
        "MessageBody" : message_body,
        "SignatureMethod" : SIGNATURE_METHOD,
        "SignatureVersion" : SIGNATURE_VERSION,
        "Timestamp" : current_date_str,
        "Version" : AWS_VERSION
    }

    #Signature should not be included in the hash, only when sending
    #Add data to the query if Signature is specified
    if option is not None:
        query_map.update(option)

    #Convert to GET query format
    return "&".join([
        f"{key}={value}"
        for key, value in query_map.items()
    ])

#Create as many signed URLs as you need
def request(credential, endpoint, data_patterns):
    url = {}
    for pattern in data_patterns:
        url[pattern] = create_presigned_url(credential, endpoint, url_encode.quote(pattern, safe = ""))
    return {
        "url" : url
    }

#Get POSTed data from APIGateway (HTTP API) arguments (data is base64 encoded on APIGateway side)
def get_payload_from_event(event):
    payload_str = ""
    if event["isBase64Encoded"]:
        payload_str = base64.b64decode(event["body"].encode(ENCODE))
    else:
        payload_str = event["body"]
    return json.loads(payload_str)

#Lambda runtime entry point
def lambda_handler(event, context):
    payload = get_payload_from_event(event)
    return {
        'statusCode': 200,
        'body': json.dumps(request(Credentials.from_iam(), Endpoint(payload["que_name"]), payload["patterns"]))
    }

# ---------------------------------------------
#The following is for local execution
#Specify when running other than Lambda
# ---------------------------------------------

#Variables for local execution
class _Default:
    def __init__(self):
        self.SQS_ACCOUNT_ID = "" #Specify the IAM access key ID
        self.SQS_SECRET_KEY = "" #Specify the IAM secret key
        self.AWS_ACCOUNT_NUMBER = "" #Specify your AWS account ID
        self.AWS_REGION = "ap-northeast-1" #Specify the region where SQS is located
        self.SQS_QUEUE_NAME = "" #Specify the destination queue name
DEFAULT = _Default()

#Entry point when running local environment debug
if __name__ == "__main__":
    print(json.dumps(request(Credentials.from_iam(), Endpoint(DEFAULT.SQS_QUEUE_NAME), [
        "Open/Open",  "Close/Open",  "Wait/Open",
        "Open/Close", "Close/Close", "Wait/Close",
        "Open/Wait",  "Close/Wait",  "Wait/Wait"
    ])))

Set the created Lambda as the back end of API Gateway.

capture_6.png

If it's OK if you move, it's okay if you read this far. After deploying API Gateway, it should work just like a "rendering".

From here on, it will be a detailed story.

Detailed story

Originally, SQS has a mechanism to send messages by GET or POST without going through AWS-SDK.

Reference: Make a query API request https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-making-api-requests.html

Anonymous user, send to public queue

The simplest request method is as follows. When you hit the URL on the reference site, a message (data) will be sent to the queue (MyQueue). [^ 2]

[^ 2]: * Version is the version of SQS. For the time being, you can send it without thinking about anything.

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue ? 
    Version = 2012-11-05 &
    Action = SendMessage &
    MessageBody = data

Note that this query will fly anonymous user messages. The default queue does not arrive with a permission error.

To receive messages in this state, you must have set them to accept public disclosure. Change the queue permissions as captured and check "Everyone".

capture_4.png

Send to a queue that allows only specific users

If you don't want to accept the public release and want to send it to a specific user, set "Who am I?".

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue ?
    Version = 2012-11-05 &
    Action = SendMessage &
    MessageBody = data &
    AWSAccessKeyId = AKIA****************

Since I set the AWSAccessKeyID, "someone" can be communicated. However, I don't know if this is really the person. You may just impersonate yourself.

Use the secret access key to prove your identity. However, since it is not possible to send the secret access key as it is, we will send the hashed version.

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue ?
    Version = 2012-11-05 &
    Action = SendMessage &
    MessageBody = data &
    AWSAccessKeyId = AKIA**************** &
    Signature = ********************************** &
    SignatureMethod = HmacSHA256 &
    SignatureVersion = 2 &
    Timestamp = 2020-04-30T10:42:54

Signature is a hash of the data to be sent, using the secret access key as a key. The data to be sent is the entire query excluding Signature and the host information combined.

Signature Method is the algorithm used to hash Signature. Since it is HMAC-SHA256, I will tell AWS about it.

Signature Version is the signature version. This time I'm using signature version 2.

TimeStamp is the hashed date and time.

Reference: Signing version 2 signing process https://docs.aws.amazon.com/ja_jp/general/latest/gr/signature-version-2.html

By the way, what is the difference from the original signed URL?

The behavior is different compared to the "signed URL" of the original S3.

** The range in which data can be changed after issuing the URL is different **

In SQS, the signature changes depending on the content of the message you send. With S3, you can send with the same signature no matter what the content of the file you send.

To send different messages with SQS, you need to issue that many Signatures.

** The lifetime of the temporary credentials passed by STS is different **

For S3, the temporary access key ID issued by Lambda can be used until the signed URL expires. Similarly for SQS, what happens if you issue and grant a temporary access key ID and secret access key?

Since the access key ID disappears when Lambda ends, authentication will not pass when Lambda returns the URL to API Gateway.

Recommended Posts

Issue a signed URL with AWS SQS
Generate a Pre-Signed URL with golang
Generate S3 signed URL with boto
Create a private repository with AWS CodeArtifact
I tried to make a url shortening service serverless with AWS CDK
Build a WardPress environment on AWS with pulumi
Try Tensorflow with a GPU instance on AWS
Build a cheap summarization system with AWS components
AWS Step Functions to learn with a sample
Issue the Amazon CloudFront Signed URL in Python
[Go] Deliver private content with CloudFront signed URL
[AWS] I made a reminder BOT with LINE WORKS
Issue S3 time-limited URL with boto3 (with file existence confirmation)
Create a Layer for AWS Lambda Python with Docker
Django + redis Issue a dedicated URL valid for 1 hour
A4 size with python-pptx
[blackbird-sqs] Monitoring AWS SQS
URL shortening with Python
AWS CDK with Python
Decorate with a decorator
I just built a virtual environment with AWS lambda layer
[AWS] I made a reminder BOT with LINE WORKS (implementation)