AWS Step Functions to learn with a sample

Step Functions announced as a workflow service. Other workflow services include Simple Workflow (SWF), but for new applications it is recommended to consider Step Functions. If it doesn't meet your needs, you can try using SWF.

There are currently 6 types of blueprints available as samples in Step functions. Just by looking at this code, it seems that you can understand the minimum specifications required.

Look at the Blueprint prepared first and create a state machine with all kinds of States defined at the end.

Blueprint Hello World スクリーンショット 2017-03-01 16.15.44.png

{
  "Comment": "A Hello World example of the Amazon States Language using a Pass state",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Pass",
      "Result": "Hello World!",
      "End": true
    }
  }
}

Wait State スクリーンショット 2017-03-01 16.16.16.png

{
  "Comment": "An example of the Amazon States Language using wait states",
  "StartAt": "FirstState",
  "States": {
    "FirstState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "Next": "wait_using_seconds"
    },
    "wait_using_seconds": {
      "Type": "Wait",
      "Seconds": 10,
      "Next": "wait_using_timestamp"
    },
    "wait_using_timestamp": {
      "Type": "Wait",
      "Timestamp": "2015-09-04T01:59:00Z",
      "Next": "wait_using_timestamp_path"
    },
    "wait_using_timestamp_path": {
      "Type": "Wait",
      "TimestampPath": "$.expirydate",
      "Next": "wait_using_seconds_path"
    },
    "wait_using_seconds_path": {
      "Type": "Wait",
      "SecondsPath": "$.expiryseconds",
      "Next": "FinalState"
    },
    "FinalState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "End": true
    }
  }
}

Retry Failure スクリーンショット 2017-03-01 16.16.43.png

{
  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "Retry": [
        {
          "ErrorEquals": ["HandledError"],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
        },
        {
          "ErrorEquals": ["States.TaskFailed"],
          "IntervalSeconds": 30,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
        },
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 5,
          "MaxAttempts": 5,
          "BackoffRate": 2.0
        }
      ],
      "End": true
    }
  }
}

Parallel スクリーンショット 2017-03-01 16.17.25.png

{
  "Comment": "An example of the Amazon States Language using a parallel state to execute two branches at the same time.",
  "StartAt": "Parallel",
  "States": {
    "Parallel": {
      "Type": "Parallel",
      "Next": "Final State",
      "Branches": [
        {
          "StartAt": "Wait 20s",
          "States": {
            "Wait 20s": {
              "Type": "Wait",
              "Seconds": 20,
              "End": true
            }
          }
        },
        {
          "StartAt": "Pass",
          "States": {
            "Pass": {
              "Type": "Pass",
              "Next": "Wait 10s"
            },
            "Wait 10s": {
              "Type": "Wait",
              "Seconds": 10,
              "End": true
            }
          }
        }
      ]
    },
    "Final State": {
      "Type": "Pass",
      "End": true
    }
  }
}

Catch Failure スクリーンショット 2017-03-01 16.18.05.png

{
  "Comment": "A Catch example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "Catch": [
        {
          "ErrorEquals": ["HandledError"],
          "Next": "CustomErrorFallback"
        },
        {
          "ErrorEquals": ["States.TaskFailed"],
          "Next": "ReservedTypeFallback"
        },
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "CatchAllFallback"
        }
      ],
      "End": true
    },
    "CustomErrorFallback": {
      "Type": "Pass",
      "Result": "This is a fallback from a custom lambda function exception",
      "End": true
    },
    "ReservedTypeFallback": {
      "Type": "Pass",
      "Result": "This is a fallback from a reserved error code",
      "End": true
    },
    "CatchAllFallback": {
      "Type": "Pass",
      "Result": "This is a fallback from a reserved error code",
      "End": true
    }
  }
}

Choice State スクリーンショット 2017-03-01 16.18.49.png

{
  "Comment": "An example of the Amazon States Language using a choice state.",
  "StartAt": "FirstState",
  "States": {
    "FirstState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "Next": "ChoiceState"
    },
    "ChoiceState": {
      "Type" : "Choice",
      "Choices": [
        {
          "Variable": "$.foo",
          "NumericEquals": 1,
          "Next": "FirstMatchState"
        },
        {
          "Variable": "$.foo",
          "NumericEquals": 2,
          "Next": "SecondMatchState"
        }
      ],
      "Default": "DefaultState"
    },

    "FirstMatchState": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OnFirstMatch",
      "Next": "NextState"
    },

    "SecondMatchState": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OnSecondMatch",
      "Next": "NextState"
    },

    "DefaultState": {
      "Type": "Fail",
      "Cause": "No Matches!"
    },

    "NextState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME",
      "End": true
    }
  }
}

combine

The following 7 types of states are available.

Create the following state machine using all 7 types of State.

スクリーンショット 2017-03-01 15.38.58.png

Implementation

JSON The above state machine is defined in JSON as below.

{
	"Comment": "Sample Step functions flow",
    "StartAt": "Process1",
    "States": {
        "Process1": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:Process1",
            "Next": "ChoiceStep"
        },
        "ChoiceStep": {
            "Type": "Choice",
            "Choices": [{
                "Variable": "$.processResult",
                "StringEquals": "Process2a",
                "Next": "SuccessProcess"
            }, {
                "Variable": "$.processResult",
                "StringEquals": "Process2b",
                "Next": "Process2b-1"
            }]
        },
        "SuccessProcess": {
            "Type": "Succeed"
        },
        "Process2b-1": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:Process2",
            "Next": "Process2b-2"
        },
        "Process2b-2": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:Process2",
          	"Catch": [{
              "ErrorEquals": [
                "HandledError"
              ],
              "Next": "FailedProcess"
            }],
            "Next": "Process3"
        },
      	"FailedProcess": {
            "Type": "Fail"      
        },
        "Process3": {
            "Type": "Wait",
            "Seconds": 30,
            "Next": "Process4"
        },
        "Process4": {
            "Type": "Parallel",
            "Next": "Process5",
            "Branches": [
                {
                    "StartAt": "Process4a",
                    "States": {
                        "Process4a": {
                            "Type": "Task",
                            "Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:Process2",
                            "End": true
                        }
                    }
                },
                {
                    "StartAt": "Process4b-1",
                    "States": {
                         "Process4b-1": {
                            "Type": "Task",
                            "Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:Process2",
                            "Retry": [{
                                "ErrorEquals": [
                                    "HandledError"
                                ],
                                "IntervalSeconds": 3,
                                "MaxAttempts": 5,
                                "BackoffRate": 1.5
                            }],
                            "Next": "Process4b-2"
                         },
                         "Process4b-2": {
                            "Type": "Pass",
                            "Result": {
                                "data1": 12345,
                                "data2": -12345
                            },
                            "ResultPath": "$.test-data",
                            "Next": "Process4b-3"
                         },
                         "Process4b-3": {
                            "Type": "Task",
                            "Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:Process2",
                            "End": true
                         }
                    }
                }
            ]
        },
        "Process5": {
          	"Type": "Task",
			"Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:Process3",
            "End": true
        }
    }
}

Lambda function

Prepare the following three Labda functions.

def lambda_handler(event, context):
    event["step"] = 1
    event["processResult"] = "Process2b"
    return event
def lambda_handler(event, context):
    event["step"] += 1
    return event
def lambda_handler(event, context):
    print event
    return event

Creation procedure

  1. Select [Step functions] from the management console.

  2. Select Get Started.

  3. Enter the following and select [Create State Machine]. Give a name to your state machine: SampleStateMachineWithVariableOfTypes

Enter the JSON from the previous one.

When you press the load button of Preview, the state of the state machine is displayed.

  1. Enter the following and select [OK].

IAM role for your state machine executions: StatesExecutionRole-us-east-1

  1. Select New Execution.
  2. Give the following value as Input.
{
    "processResult": "Process2b"
}
  1. Select Start Execution.

output

Output

output: [{"processResult": "Process2b", "step": 4},{"test-data": {"data1": 12345, "data2": -12345}, "processResult": "Process2b", "step": 5}]

Tips -You can check the correctness of the syntax with statelint. (I feel that there are times when it is inaccurate.) --The correctness of JSON syntax can also be confirmed with the jq command. --You can check the syntax by pressing the refresh button next to Preview. --There may be an error even after selecting [Create State Machine] where the above syntax check does not apply, so you can check it by pressing it before execution. --Debugging Lambda creation It is easy to make trial and error if you can debug with python sample.py as follows.

def lambda_handler(event, context):
    event["step"] = 1
    event["processResult"] = "Process2b"
    return event

if __name__ == "__main__":
    lambda_handler("","")

Caution

--I get angry when I define extra tasks

reference

Recommended Posts

AWS Step Functions to learn with a sample
[AWS] Play with Step Functions (SAM + Lambda) Part.3 (Branch)
A sample to try Factorization Machines quickly with fastFM
[AWS] Play with Step Functions (SAM + Lambda) Part.1 (Basic)
Delete DynamoDB data after 5 minutes with AWS Step Functions
[AWS] Play with Step Functions (SAM + Lambda) Part.2 (Parameter)
Learn librosa with a tutorial 1
Python sample to learn XOR with genetic algorithm with neural network
The first step to creating a serverless application with Zappa
[Python] The first step to making a game with Pyxel
Data engineers learn DevOps with a view to MLOps. ① Getting started
How to create a serverless machine learning API with AWS Lambda
Learn to colorize monochrome images with Chainer
How to add a package with PyCharm
Issue a signed URL with AWS SQS
I want to play with aws with python
"How to pass PATH" to learn with homebrew
How to use CUT command (with sample)
Connect to s3 with AWS Lambda Python
Sample program to display video with PyQt
Create a private repository with AWS CodeArtifact
I tried to make a url shortening service serverless with AWS CDK
Preparing to learn technical indicators with TFlearn
Sample to convert image to Wavelet with Python
When I tried to make a VPC with AWS CDK but couldn't make it
[AWS] Wordpress How to deal with "The response is not a correct JSON response"
I tried to create a model with the sample of Amazon SageMaker Autopilot
We built a mechanism to download a large zip file with basic authentication with aws
How to create sample CSV data with hypothesis
How to read a CSV file with Python 2/3
Send a message to LINE with Python (LINE Notify)
[Computer vision] Epipolar geometry to learn with cats
How to send a message to LINE with curl
A memorandum to make WebDAV only with nginx
OpenAI Gym to learn with PD-controlled Cart Pole
Build a WardPress environment on AWS with pulumi
How to draw a 2-axis graph with pyplot
I tried to learn logical operations with TF Learn
Try Tensorflow with a GPU instance on AWS
Learn to recognize handwritten numbers (MNIST) with Caffe
How to develop a cart app with Django
Randomly sample MNIST data to create a dataset
Try to draw a life curve with python
How to make a dictionary with a hierarchical structure.
I want to make a game with Python
Try to make a "cryptanalysis" cipher with Python
Pretend to be a server with two PCs
Decide to assign a laboratory with Python (fiction)
Easily log in to AWS with multiple accounts
Sample to send slack notification with python lambda
Sample to draw a simple clock using ebiten
Steps to create a Twitter bot with python
Try to make a dihedral group with Python
How to Learn Kaldi with the JUST Corpus
[Dedicated to Telewa! ] PC operation with a webcam
I want to write to a file with Python
[PyTorch] Sample ⑧ ~ How to build a complex model ~
A layman wants to get started with Python
How to use fixture in Django to populate sample data associated with a user model
Quickly create a Python data analysis dashboard with Streamlit and deploy it to AWS
[AWS] Development environment version that tried to build a Python environment with eb [Elastic Beanstalk]