AWS Lambda + Step Functions

Why Lambda + Step Functions

Pairing Lambda with Step Functions gives you managed, serverless compute and a visual workflow engine with retries, branching, and parallelism. You write minimal glue code, push orchestration into state definitions, and gain robust error handling without rolling your own state machine.

• Separation of concerns: Lambdas focus on business logic; Step Functions own orchestration and retries.
• Elastic by default: Scale to thousands of parallel tasks with Map and Distributed Map.
• Pay-per-use: Cost tracks invocations and duration; no servers to manage.
• Built-in reliability: Declarative Retry, Catch, Timeout, and Heartbeat.

Reference Architecture

The minimal pattern: Lambda A (Invoker) starts an Express state machine using StartSyncExecution. The state machine invokes Lambda B and returns its result to Lambda A, which then replies to the caller (e.g., API Gateway).

Client/API → Lambda A → StartSyncExecution → Step Functions (Express)
                               │
                               └─ LambdaInvoke → Lambda B → result → Step Functions → Lambda A → response

Mini Project (All-Python)

Copy these files into a folder (e.g., lambda-sfn-python/). Run CDK to deploy. The stack creates two Lambdas (Invoker / Worker) and an Express state machine that synchronously returns the worker result.

CDK Stack (Python)

from constructs import Construct
import aws_cdk as cdk
from aws_cdk import (
    Duration,
    aws_lambda as _lambda,
    aws_stepfunctions as sfn,
    aws_stepfunctions_tasks as tasks,
)
from pathlib import Path

LAMBDA_DIR = str(Path(__file__).parent.joinpath("lambda").resolve())

class LambdaSfnPythonStack(cdk.Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        worker_fn = _lambda.Function(
            self, "WorkerFn",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="lambda_b.handler",
            code=_lambda.Code.from_asset(LAMBDA_DIR),
            memory_size=256,
            timeout=Duration.seconds(10),
        )

        invoke_worker = tasks.LambdaInvoke(
            self, "InvokeWorker",
            lambda_function=worker_fn,
            payload_response_only=True,
        )

        state_machine = sfn.StateMachine(
            self, "DemoExpressStateMachine",
            definition_body=sfn.DefinitionBody.from_chainable(invoke_worker),
            state_machine_type=sfn.StateMachineType.EXPRESS,
            tracing_enabled=True,
        )

        invoker_fn = _lambda.Function(
            self, "InvokerFn",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="lambda_a.handler",
            code=_lambda.Code.from_asset(LAMBDA_DIR),
            memory_size=256,
            timeout=Duration.seconds(10),
            environment={"STATE_MACHINE_ARN": state_machine.state_machine_arn},
        )

        state_machine.grant_start_sync_execution(invoker_fn)

Lambda A (Invoker)

import json, os, uuid
from datetime import datetime
import boto3

SFN_ARN = os.environ["STATE_MACHINE_ARN"]
sfn = boto3.client("stepfunctions")

def handler(event, context):
    payload = {
        "correlationId": str(uuid.uuid4()),
        "receivedAt": datetime.utcnow().isoformat() + "Z",
        "request": event,
    }
    resp = sfn.start_sync_execution(stateMachineArn=SFN_ARN, input=json.dumps(payload))
    out = resp.get("output")
    return {
        "ok": resp.get("status") == "SUCCEEDED",
        "result": json.loads(out) if out else None,
    }

Lambda B (Worker)

def handler(event, context):
    req = (event or {}).get("request", {})
    op = req.get("operation", "echo")
    n = req.get("number", 0)
    if op == "double":
        val, msg = n * 2, f"doubled {n} -> {n*2}"
    elif op == "square":
        val, msg = n * n, f"squared {n} -> {n*n}"
    else:
        val, msg = req, "echo"
    return {"operation": op, "inputNumber": n, "value": val, "message": msg}

Common Use Cases

• Event-driven ETL: Ingest → transform → validate → load, with fan-out via Map.
• API composition: Aggregate multiple downstream calls with retries and timeouts.
• Human-in-the-loop: Approval steps, wait states, and callbacks using task tokens (Standard).
• ML pipelines: Pre/post-processing around SageMaker jobs or Batch tasks.
• Back-office ops: Scheduled workflows, report generation, batch reconciliation.

Benefits

• Fast iteration: Small, focused functions and declarative state machines.
• Resilience built-in: Retry/Catch/backoff without custom frameworks.
• Parallelism: Map and Distributed Map scale out work safely.
• Observability: Execution history + CloudWatch Logs + X-Ray traces.

Drawbacks / Pitfalls

• State history cost & size: Large payloads increase cost and may hit limits. Store blobs in S3 and pass references.
• Cold starts / latency: Latency-sensitive APIs may require Provisioned Concurrency or lighter runtimes.
• VPC overhead: Only attach a VPC when needed (private DBs). Consider VPC endpoints to avoid NAT costs.
• Long-running work: Prefer Standard type or move heavy compute to ECS/EKS/Batch with callbacks.
• Fan-out pressure: Cap concurrency to protect downstreams and add DLQs.

Things to Consider

• Limits: Payload sizes, history retention, execution duration differ between Express and Standard.
• Idempotency: Retries happen—design safe, idempotent handlers.
• Partial batch success: For SQS/Kinesis sources, enable partial responses to avoid reprocessing successes.
• Schema & contracts: Version your inputs/outputs; validate early.
• Security: Least-privilege IAM, parameterize secrets with Secrets Manager/SSM.

Best Practices & Cost

Lambda

• Right-size memory (CPU scales with memory). Benchmark a few points for fastest-cheapest mix.
• Trim dependencies; lazy-init heavy SDKs. Cache clients outside handlers.
• Use Provisioned Concurrency or SnapStart (runtime support varies) for consistent latency.
• Increase /tmp only when needed (512MB→10GB).

Step Functions

• Prefer service integrations (DynamoDB, S3, EventBridge, ECS) over Lambda glue.
• Control payload growth with ResultPath, Parameters, and ResultSelector.
• Use Map/Distributed Map for parallelism; cap concurrency.
• Pick Express for high-throughput short tasks; Standard for long-lived workflows.

Observability, Testing, Security

• Logging/Tracing: Structured logs, X-Ray traces across Lambda and Step Functions.
• Metrics/SLOs: Emit p95 latency, error rate, cost per unit, DLQ depth. Alarm on anomalies.
• Testing: Unit-test handlers; integration-test state definitions via LocalStack or Step Functions local.
• Security: Fine-grained IAM; encrypt at rest (KMS) and in-transit (TLS). Rotate secrets.

Deploy & Run

aws-cdk-lib==2.154.0
constructs>=10.0.0,<11.0.0

import aws_cdk as cdk
from lambda_sfn_python_stack import LambdaSfnPythonStack
app = cdk.App()
LambdaSfnPythonStack(app, "LambdaSfnPythonStack")
app.synth()

Bootstrap once per account/region with cdk bootstrap, then cdk deploy. Invoke the invoker Lambda with a payload like:

aws lambda invoke   --function-name <InvokerFnName>   --payload '{"operation":"double","number":21}'   --cli-binary-format raw-in-base64-out   response.json && cat response.json

AWS Step Functions Components

AWS Step Functions orchestrates workflows with a declarative JSON spec calledAmazon States Language (ASL). Below are the core building blocks you’ll use to design, run, and operate your workflows.

State Machine

The top-level workflow definition written in ASL. It declares the StartAt state, the full set of States, and (optionally) TimeoutSeconds, Tracing, etc.

{
  "Comment": "Example workflow",
  "StartAt": "DoWork",
  "States": {
    "DoWork": { "Type": "Pass", "Result": "ok", "End": true }
  }
}

States

States are the steps in your workflow. Common types:

Task State

Performs work by calling an AWS service (service integration) or an external worker (Activity). Use Retry/Catch for resilience, and TimeoutSeconds/HeartbeatSeconds to detect hangs.

{
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke",
  "Parameters": {
    "FunctionName": "arn:aws:lambda:REGION:ACCOUNT:function:WorkerFn",
    "Payload.$": "$"              // pass whole input; use ResultSelector/Path to trim
  },
  "ResultPath": "$.worker",
  "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 }],
  "Catch":  [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "FailState" }],
  "Next": "NextState"
}

Choice State

Adds branching logic using JSONPath conditions.

{
  "Type": "Choice",
  "Choices": [
    { "Variable": "$.request.operation", "StringEquals": "double", "Next": "Double" },
    { "Variable": "$.request.operation", "StringEquals": "square", "Next": "Square" }
  ],
  "Default": "Echo"
}

Parallel State

Runs multiple branches concurrently. Execution continues when all (or the first, if configured) branches complete.

{
  "Type": "Parallel",
  "Branches": [
    { "StartAt": "A1", "States": { "A1": { "Type": "Pass", "Result": "A", "End": true } } },
    { "StartAt": "B1", "States": { "B1": { "Type": "Pass", "Result": "B", "End": true } } }
  ],
  "ResultPath": "$.parallel",
  "Next": "NextState"
}

Map State

Iterates over an array. Two modes: Inline Map (bounded fan-out) and Distributed Map (massive scale; items can come from S3). Use MaxConcurrency to protect downstreams.

{
  "Type": "Map",
  "ItemsPath": "$.items",
  "MaxConcurrency": 40,
  "Iterator": {
    "StartAt": "ProcessItem",
    "States": {
      "ProcessItem": {
        "Type": "Task",
        "Resource": "arn:aws:states:::lambda:invoke",
        "Parameters": { "FunctionName": "WorkerFn", "Payload.$": "$" },
        "End": true
      }
    }
  },
  "ResultPath": "$.processed",
  "Next": "NextState"
}

Pass State

Passes input to output without work. Useful for shaping data.

{ "Type": "Pass", "Parameters": { "status": "ok", "input.$": "$" }, "ResultPath": "$.meta", "Next": "NextState" }

Wait State

Pauses execution by duration or until a specific timestamp.

{ "Type": "Wait", "Seconds": 30, "Next": "NextState" }

Succeed / Fail State

Explicitly end an execution successfully or as a failure.

{ "Type": "Succeed" }   // or   { "Type": "Fail", "Error": "MyError", "Cause": "Message" }

Activity Tasks (external workers)

For work performed by non-AWS systems that poll Step Functions. Prefer service integrations or Lambda when possible.

{ "Type": "Task", "Resource": "arn:aws:states:REGION:ACCOUNT:activity:MyActivity", "Next": "NextState" }

Executions

Each run of a state machine is an Execution with its own input, history, and output. Express supports StartSyncExecution (returns output inline). Standard supports long-running workflows and rich history.

Amazon States Language (ASL)

JSON language for declaring states, transitions, and error handling. Key fields: Type, Next/End, Parameters, ResultPath,ResultSelector, Retry, Catch, TimeoutSeconds, HeartbeatSeconds.

{
  "Comment": "Lambda A -> Step Functions -> Lambda B",
  "StartAt": "Validate",
  "States": {
    "Validate": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.request.operation", "StringEquals": "double", "Next": "Work" },
        { "Variable": "$.request.operation", "StringEquals": "square", "Next": "Work" }
      ],
      "Default": "Echo"
    },
    "Work": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "WorkerFn", "Payload.$": "$" },
      "ResultPath": "$.result",
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 1, "MaxAttempts": 3, "BackoffRate": 2 }],
      "Catch":  [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "Failed" }],
      "Next": "Success"
    },
    "Echo":   { "Type": "Pass", "ResultPath": "$.result", "Next": "Success" },
    "Success":{ "Type": "Succeed" },
    "Failed": { "Type": "Fail", "Error": "Unhandled", "Cause": "See $.error" }
  }
}

AWS Service Integrations

Direct integrations (no Lambda needed) reduce latency and cost: DynamoDB, S3, EventBridge, Step Functions, SageMaker, ECS, Batch, Glue, SNS/SQS, Bedrock, and more. Prefer these over Lambda “glue” when possible.

Glossary

Below is a consolidated glossary of AWS Lambda and Step Functions concepts — followed by a visual summary table for quick scanning. This helps engineers and architects recall each key component, its purpose, and how it fits into real workflows.

• Amazon States Language (ASL): The JSON-based language used to define Step Functions workflows, specifying states, transitions, input/output paths, retries, and error handling.
• State Machine: The complete definition of a workflow in Step Functions, made up of individual states and transitions.
• State: A single step in a workflow, such as a task, choice, wait, or success state.
• Task State: Executes a specific unit of work, like invoking a Lambda function or an AWS service integration.
• Choice State: Adds branching logic, directing the workflow based on evaluated conditions.
• Parallel State: Runs multiple branches of execution concurrently, merging results afterward.
• Map / Distributed Map: Iterates over an array or S3 dataset to run sub-state machines for each item.
• Pass State: Passes its input directly to output — often for restructuring or debugging workflows.
• Wait State: Pauses execution until a given time or for a specific duration.
• Succeed / Fail State: Explicitly ends an execution successfully or marks it as failed.
• Retry Policy / Catch: Defines how to handle transient failures or reroute on specific errors.
• Parameters / ResultPath / ResultSelector: Control how data flows between states, reducing payload size.
• StartExecution / StartSyncExecution: APIs to start asynchronous or synchronous executions, respectively.
• Task Token: A callback token allowing external systems or humans to resume paused workflows.
• Activity Task: Represents external work units performed by non-AWS workers polling Step Functions.
• Execution ARN: A globally unique identifier for each execution run.
• Execution History: Timeline log of every state transition and result within a workflow.
• Integration Patterns: How Step Functions communicate with AWS services — Request/Response, Wait-for-Callback, or Fire-and-Forget.
• Express Workflow: Optimized for short-lived, high-throughput workflows with millisecond billing.
• Standard Workflow: Long-running, durable workflows supporting detailed history and manual steps.
• IAM Role for Step Functions: Grants Step Functions permission to invoke services or Lambdas.
• Context Object ($$): Provides metadata like execution name, state name, timestamps, and retry counts.
• Dynamic References: Injects runtime values (e.g., Secrets Manager parameters) into state definitions.
• CloudWatch Logs & Metrics: Default observability layer for Step Functions and Lambda executions.
• X-Ray Tracing: Enables distributed tracing to visualize service dependencies and latency.
• DLQ (Dead Letter Queue): Captures failed messages or executions for later analysis.
• Idempotency: Designing states to handle retries without duplicating side effects.
• Execution Output: The final JSON returned by a completed execution.
• Workflow Chaining: Linking multiple state machines for modular orchestration.
• Callback Pattern: A workflow pattern where Step Functions waits for an external process to respond.
• Human Approval Step: A manual checkpoint awaiting a user decision before continuing.
• Cost Model: Step Functions cost depends on state transitions (Standard) or duration (Express).
• Provisioned Concurrency: Keeps Lambda warm for low-latency requests.
• State Transition: Each move from one state to another — a key billing metric for Standard workflows.
• Audit & Compliance: Execution logs provide tamper-proof evidence trails for governance or security reviews.