Skip to main content

Cloud development often involves the challenge of testing applications against real cloud services like AWS, which can be slow, costly, and complex. LocalStack and Docker provide a powerful solution by enabling you to run a fully functional local AWS cloud stack on your development machine. In this guide, we'll explore how to set up and use LocalStack with Docker to create an efficient local cloud development environment.

Local Cloud Development with Docker and LocalStack: Emulate AWS Services Locally

Table of Contents #

Introduction to Local Cloud Development #

Developing applications for cloud environments presents unique challenges. Traditional approaches require deploying to actual cloud services for testing, which introduces delays, costs, and complexities. This guide is based on our Lab8 LocalStack Example from the Docker Practical Guide repository.

Local cloud development aims to solve these problems by:

  • Creating a local environment that mimics cloud services
  • Enabling rapid iteration without remote deployments
  • Eliminating costs associated with cloud resources during development
  • Allowing offline development and testing
  • Simplifying the development experience

Understanding LocalStack #

LocalStack is an open-source tool that provides a local, fully functional AWS cloud stack. It allows you to develop and test cloud applications without connecting to real AWS services.

┌───────────────────────────────────────────────────────────┐
│                       LocalStack                          │
│                                                           │
│  ┌─────────────┐   ┌─────────────┐   ┌─────────────┐      │
│  │             │   │             │   │             │      │
│  │  S3 Buckets │   │   Lambda    │   │    SQS      │      │
│  │             │   │  Functions  │   │   Queues    │      │
│  └─────────────┘   └─────────────┘   └─────────────┘      │
│                                                           │
│  ┌─────────────┐   ┌─────────────┐   ┌─────────────┐      │
│  │             │   │             │   │             │      │
│  │  DynamoDB   │   │  API Gateway│   │   SNS       │      │
│  │   Tables    │   │             │   │   Topics    │      │
│  └─────────────┘   └─────────────┘   └─────────────┘      │
│                                                           │
│  ┌─────────────┐   ┌─────────────┐   ┌─────────────┐      │
│  │             │   │             │   │             │      │
│  │  Kinesis    │   │ CloudWatch  │   │  And More   │      │
│  │  Streams    │   │             │   │             │      │
│  └─────────────┘   └─────────────┘   └─────────────┘      │
│                                                           │
└───────────────────────────────────────────────────────────┘

LocalStack supports a wide range of AWS services, including:

  • Storage: S3, DynamoDB, ElastiCache
  • Compute: Lambda, ECS, EC2
  • Messaging: SQS, SNS, EventBridge
  • Databases: RDS, DynamoDB, ElastiCache
  • API Services: API Gateway, AppSync
  • And many more: CloudFormation, IAM, CloudWatch, etc.

With LocalStack, you can use the same AWS SDKs, CLI commands, and infrastructure-as-code tools (like AWS CDK, CloudFormation, or Terraform) that you would use with real AWS services, but everything runs locally on your machine.

Setting Up Docker with LocalStack #

Let's start by setting up a local environment with Docker and LocalStack:

Docker Compose Configuration #

Create a docker-compose.yml file that includes LocalStack and your application services:

version: "3.8"

services:
localstack:
image: localstack/localstack:latest
ports:
- "4566:4566" # LocalStack endpoint
- "4510-4559:4510-4559" # external services port range
environment:
- DEBUG=1
- DOCKER_HOST=unix:///var/run/docker.sock
- SERVICES=s3,lambda,dynamodb,apigateway,iam,sqs,sns
- DEFAULT_REGION=us-east-1
- AWS_DEFAULT_REGION=us-east-1
- HOSTNAME_EXTERNAL=localstack
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "${PWD}/.localstack:/tmp/localstack"

frontend:
build: ./frontend
ports:
- "8080:80"
depends_on:
- nginx

backend:
build: ./backend
environment:
- AWS_ENDPOINT=http://localstack:4566
- AWS_REGION=us-east-1
- AWS_ACCESS_KEY_ID=test
- AWS_SECRET_ACCESS_KEY=test
depends_on:
- localstack

nginx:
build: ./nginx
ports:
- "80:80"
depends_on:
- backend
- localstack

This Docker Compose file sets up:

  1. LocalStack: The core service that emulates AWS
  2. Frontend: A React application that interacts with the backend
  3. Backend: A Node.js API that communicates with AWS services (through LocalStack)
  4. Nginx: A reverse proxy that routes requests to the appropriate services

Application Architecture #

Our example implements a modern microservices architecture:

┌───────────────────────────────────────────────────────────────────────────┐
│                                Docker Compose                             │
│                                                                           │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌──────────┐ │
│  │    Nginx    │     │   Frontend  │     │   Backend   │     │LocalStack│ │
│  │   Proxy     │     │    React    │     │   Node.js   │     │          │ │
│  │  Container  │     │  Container  │     │  Container  │     │ Container│ │
│  └──────┬──────┘     └──────┬──────┘     └──────┬──────┘     └────┬─────┘ │
│         │                   │                   │                 │       │
│         │                   │                   │                 │       │
│         │  ┌───────────────┐│                   │                 │       │
│         └──┤ /             ├┘                   │                 │       │
│            └───────────────┘                    │                 │       │
│                                                 │                 │       │
│            ┌───────────────┐                    │                 │       │
│         ┌──┤ /api          ├────────────────────┘                 │       │
│         │  └───────────────┘                                      │       │
│         │                                                         │       │
│         │                                      ┌─────────────┐    │       │
│         │                                      │ AWS Services│    │       │
│         └──────────────────────────────────────┤ Emulation   ├────┘       │
│                                                └─────────────┘            │
│                                                                           │
└───────────────────────────────────────────────────────────────────────────┘
                                    ▲
                                    │
                                    │
                                    │
                                    ▼
                              ┌──────────┐
                              │  Client  │
                              │ Browser  │
                              └──────────┘

Creating a Microservices Application #

Our example application consists of several components:

1. Backend Service #

The Node.js backend service interacts with AWS services through LocalStack:

// backend/server.js
const express = require("express");
const AWS = require("aws-sdk");
const bodyParser = require("body-parser");

const app = express();
app.use(bodyParser.json());

// Configure AWS to use LocalStack
const endpoint = process.env.AWS_ENDPOINT || "http://localhost:4566";
const region = process.env.AWS_REGION || "us-east-1";

AWS.config.update({
endpoint,
region,
accessKeyId: "test",
secretAccessKey: "test",
});

// Create service clients
const sqs = new AWS.SQS();
const queueUrl = `${endpoint}/000000000000/message-queue`;

// API endpoints
app.get("/api/health", (req, res) => {
res.json({ status: "healthy", awsEndpoint: endpoint });
});

app.post("/api/messages", async (req, res) => {
try {
const { message } = req.body;
const params = {
QueueUrl: queueUrl,
MessageBody: JSON.stringify({
text: message,
timestamp: new Date().toISOString(),
}),
};

const result = await sqs.sendMessage(params).promise();
res.json({ success: true, messageId: result.MessageId });
} catch (error) {
console.error("Error sending message:", error);
res.status(500).json({ error: error.message });
}
});

app.get("/api/messages", async (req, res) => {
try {
const params = {
QueueUrl: queueUrl,
MaxNumberOfMessages: 10,
WaitTimeSeconds: 1,
};

const result = await sqs.receiveMessage(params).promise();
const messages = result.Messages || [];

// Process and delete received messages
const processedMessages = await Promise.all(
messages.map(async (msg) => {
const body = JSON.parse(msg.Body);

// Delete the message from the queue
await sqs
.deleteMessage({
QueueUrl: queueUrl,
ReceiptHandle: msg.ReceiptHandle,
})
.promise();

return body;
})
);

res.json({ messages: processedMessages });
} catch (error) {
console.error("Error receiving messages:", error);
res.status(500).json({ error: error.message });
}
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
console.log(`Backend service running on port ${PORT}`);
});

2. Frontend Service #

A simple React application that communicates with the backend:

// frontend/src/App.js
import React, { useState, useEffect } from "react";
import "./App.css";

function App() {
const [message, setMessage] = useState("");
const [messages, setMessages] = useState([]);
const [status, setStatus] = useState("Loading...");

useEffect(() => {
fetchHealthStatus();
fetchMessages();

// Poll for new messages every 5 seconds
const interval = setInterval(fetchMessages, 5000);
return () => clearInterval(interval);
}, []);

const fetchHealthStatus = async () => {
try {
const response = await fetch("/api/health");
const data = await response.json();
setStatus(
`Backend status: ${data.status}, AWS endpoint: ${data.awsEndpoint}`
);
} catch (error) {
setStatus("Error connecting to backend");
}
};

const fetchMessages = async () => {
try {
const response = await fetch("/api/messages");
const data = await response.json();
setMessages(data.messages);
} catch (error) {
console.error("Error fetching messages:", error);
}
};

const sendMessage = async () => {
if (!message.trim()) return;

try {
await fetch("/api/messages", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ message }),
});
setMessage("");
fetchMessages();
} catch (error) {
console.error("Error sending message:", error);
}
};

return (
<div className="App">
<header className="App-header">
<h1>LocalStack SQS Demo</h1>
<p className="status">{status}</p>

<div className="message-form">
<input
type="text"
value={message}
onChange={(e) => setMessage(e.target.value)}
placeholder="Enter a message"
/>
<button onClick={sendMessage}>Send</button>
</div>

<div className="messages">
<h2>Messages</h2>
{messages.length === 0 ? (
<p>No messages yet</p>
) : (
<ul>
{messages.map((msg, index) => (
<li key={index}>
<strong>{new Date(msg.timestamp).toLocaleString()}</strong>:{" "}
{msg.text}
</li>
))}
</ul>
)}
</div>
</header>
</div>
);
}

export default App;

3. Nginx Configuration #

A reverse proxy to route requests between services:

# nginx/nginx.conf
server {
listen 80;

location / {
proxy_pass http://frontend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}

location /api/ {
proxy_pass http://backend:3000/api/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}

# LocalStack UI access (if needed)
location /localstack/ {
proxy_pass http://localstack:8080/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}

LocalStack Configuration and Environment #

To effectively use LocalStack, you need to configure your environment properly:

Environment Variables #

# AWS credentials for LocalStack (any values work)
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# Point AWS SDKs to LocalStack
export AWS_ENDPOINT_URL=http://localhost:4566

AWS CLI Configuration #

Create a profile for LocalStack in your AWS config:

aws configure set aws_access_key_id test --profile localstack
aws configure set aws_secret_access_key test --profile localstack
aws configure set region us-east-1 --profile localstack
aws configure set endpoint_url http://localhost:4566 --profile localstack

Now you can use the --profile localstack flag with AWS CLI commands:

aws --profile localstack s3 ls
aws --profile localstack sqs list-queues

Deploying Infrastructure with AWS CDK #

AWS Cloud Development Kit (CDK) allows you to define cloud infrastructure using familiar programming languages. With LocalStack, you can deploy CDK stacks locally:

CDK Stack Definition #

// cdk/lib/message-queue-stack.ts
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as sqs from "aws-cdk-lib/aws-sqs";

export class MessageQueueStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);

// Create an SQS queue
const queue = new sqs.Queue(this, "MessageQueue", {
queueName: "message-queue",
visibilityTimeout: cdk.Duration.seconds(30),
retentionPeriod: cdk.Duration.days(7),
});

// Output the queue URL for reference
new cdk.CfnOutput(this, "QueueUrl", {
value: queue.queueUrl,
description: "The URL of the Message Queue",
});
}
}

LocalStack Deployment Script #

Create a script to deploy the infrastructure to LocalStack:

#!/bin/bash
# local-deploy.sh

# Set environment variables for LocalStack
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1
export CDK_DEPLOY_ACCOUNT=000000000000
export CDK_DEPLOY_REGION=us-east-1
export LOCALSTACK_HOSTNAME=localhost

echo "Waiting for LocalStack to be ready..."
until curl -s http://localhost:4566/_localstack/health | grep -q '"s3": "running"'; do
sleep 2
done
echo "LocalStack is ready!"

echo "Deploying infrastructure to LocalStack..."
cd cdk && npm install
npx cdk bootstrap --toolkit-stack-name cdk-toolkit aws://000000000000/us-east-1 --endpoint-url http://localhost:4566
npx cdk deploy --all --require-approval never --endpoint-url http://localhost:4566

echo "Infrastructure deployed successfully!"

Testing Against Emulated Services #

LocalStack allows you to test your application against emulated AWS services:

Interacting with SQS #

// Example: Sending a message to SQS
const AWS = require("aws-sdk");
const sqs = new AWS.SQS({
endpoint: "http://localhost:4566",
region: "us-east-1",
accessKeyId: "test",
secretAccessKey: "test",
});

async function sendMessage(message) {
const params = {
QueueUrl: "http://localhost:4566/000000000000/message-queue",
MessageBody: JSON.stringify(message),
};

try {
const result = await sqs.sendMessage(params).promise();
console.log("Message sent:", result.MessageId);
return result;
} catch (error) {
console.error("Error sending message:", error);
throw error;
}
}

// Example: Receiving messages from SQS
async function receiveMessages() {
const params = {
QueueUrl: "http://localhost:4566/000000000000/message-queue",
MaxNumberOfMessages: 10,
WaitTimeSeconds: 1,
};

try {
const result = await sqs.receiveMessage(params).promise();
const messages = result.Messages || [];
console.log(`Received ${messages.length} messages`);
return messages;
} catch (error) {
console.error("Error receiving messages:", error);
throw error;
}
}

Storage with S3 #

// Example: Working with S3 buckets
const AWS = require("aws-sdk");
const s3 = new AWS.S3({
endpoint: "http://localhost:4566",
region: "us-east-1",
accessKeyId: "test",
secretAccessKey: "test",
s3ForcePathStyle: true, // Required for LocalStack
});

async function createBucket(bucketName) {
try {
await s3.createBucket({ Bucket: bucketName }).promise();
console.log(`Bucket created: ${bucketName}`);
} catch (error) {
console.error("Error creating bucket:", error);
throw error;
}
}

async function uploadFile(bucketName, key, body) {
try {
await s3
.putObject({
Bucket: bucketName,
Key: key,
Body: body,
})
.promise();
console.log(`File uploaded: ${key}`);
} catch (error) {
console.error("Error uploading file:", error);
throw error;
}
}

Development Workflow Improvements #

Using LocalStack and Docker together significantly improves the development workflow:

1. Faster Development Cycles #

Without LocalStack, the development cycle looks like this:

┌───────────┐     ┌────────────┐     ┌─────────────┐     ┌───────────┐
│           │     │            │     │             │     │           │
│  Write    │────►│  Deploy to │────►│ Test        │────►│ Debug &   │
│  Code     │     │  AWS       │     │             │     │ Iterate   │
│           │     │            │     │             │     │           │
└───────────┘     └────────────┘     └─────────────┘     └───────────┘
                        │                                      │
                        │                                      │
                        ▼                                      │
                  ┌────────────┐                               │
                  │            │                               │
                  │   Wait     │                               │
                  │            │                               │
                  └────────────┘                               │
                        │                                      │
                        └──────────────────────────────────────┘

With LocalStack, the cycle becomes much faster:

┌───────────┐     ┌────────────┐     ┌───────────┐
│           │     │            │     │           │
│  Write    │────►│  Test with │────►│ Debug &   │
│  Code     │     │ LocalStack │     │ Iterate   │
│           │     │            │     │           │
└───────────┘     └────────────┘     └───────────┘
      ▲                                   │
      │                                   │
      └───────────────────────────────────┘

2. Cost Savings #

LocalStack eliminates the costs associated with:

  • Running cloud services during development
  • Data transfer for frequent deployments
  • Mistakes in development environments
  • Unused or forgotten resources

3. Offline Development #

With LocalStack, you can develop and test without an internet connection, which is ideal for:

  • Working while traveling
  • Environments with limited connectivity
  • Scenarios where AWS access is restricted

4. Consistent Testing Environment #

LocalStack provides a fresh, consistent environment for each test run, which helps:

  • Eliminate test flakiness from external dependencies
  • Ensure reproducible test results
  • Run tests in parallel without interference

Common Challenges and Solutions #

When working with LocalStack, you might encounter a few challenges:

1. Service Parity Issues #

Challenge: LocalStack Community Edition doesn't support all AWS services or features.

Solution:

  • Check the LocalStack documentation for supported services
  • Consider LocalStack Pro for more comprehensive service coverage
  • Implement fallbacks or mocks for unsupported services

2. Configuration Differences #

Challenge: Some configurations work differently between LocalStack and real AWS.

Solution:

  • Use environment variables to switch endpoints based on environment
  • Create abstraction layers for service interactions
  • Document known differences in your project
// Example abstraction for SQS
class QueueService {
constructor() {
const isLocalEnvironment = process.env.NODE_ENV === "development";

this.sqs = new AWS.SQS({
endpoint: isLocalEnvironment ? "http://localhost:4566" : undefined,
region: "us-east-1",
// Other configs based on environment
});

this.queueUrl = isLocalEnvironment
? "http://localhost:4566/000000000000/message-queue"
: "https://sqs.us-east-1.amazonaws.com/123456789012/message-queue";
}

async sendMessage(message) {
// Implementation remains the same
}

async receiveMessages() {
// Implementation remains the same
}
}

3. Docker Resource Usage #

Challenge: Running LocalStack alongside other containers can be resource-intensive.

Solution:

  • Configure container resource limits in Docker Compose
  • Use selective service activation in LocalStack
  • Consider running resource-intensive components on-demand

4. Persistence Between Restarts #

Challenge: LocalStack state is lost when containers restart.

Solution:

  • Use volume mounts to persist LocalStack data
  • Implement infrastructure-as-code to recreate resources
  • Add initialization scripts to your startup process

Production vs. Local Development #

While LocalStack is excellent for development, it's important to understand the differences between local and production environments:

Key Differences #

┌───────────────────────────────────────────────────────────┐
│           LocalStack vs. AWS Production                   │
│                                                           │
│  ┌────────────────┐          ┌────────────────┐           │
│  │                │          │                │           │
│  │   LocalStack   │          │   AWS Cloud    │           │
│  │                │          │                │           │
│  │ - Limited      │          │ - Complete     │           │
│  │   service      │          │   service      │           │
│  │   parity       │          │   coverage     │           │
│  │                │          │                │           │
│  │ - Simplified   │          │ - Full IAM and │           │
│  │   security     │          │   security     │           │
│  │                │          │                │           │
│  │ - Single host  │          │ - Distributed  │           │
│  │   performance  │          │   performance  │           │
│  │                │          │                │           │
│  │ - Free and     │          │ - Usage-based  │           │
│  │   local        │          │   pricing      │           │
│  │                │          │                │           │
│  └────────────────┘          └────────────────┘           │
│                                                           │
└───────────────────────────────────────────────────────────┘

Transitioning to Production #

To ensure a smooth transition from LocalStack to real AWS:

  1. Use Infrastructure as Code: Define resources using CDK, CloudFormation, or Terraform that works with both LocalStack and AWS
  2. Implement Feature Flags: Allow switching between local and cloud services
  3. Maintain Environment Parity: Keep local and cloud configurations as similar as possible
  4. Automated Testing: Run tests against both environments
  5. Gradual Migration: Test with real AWS services before full deployment

Cleanup #

After working with LocalStack for local cloud development, it's important to properly clean up your environment. This includes stopping and removing Docker containers, cleaning up AWS resources within LocalStack, and removing any local files created during development.

First, stop and remove the Docker containers:

# Stop and remove all containers defined in docker-compose.yml
docker compose down

# Include volumes if you want to completely reset your LocalStack state
docker compose down -v

# If you have multiple compose files, specify them
docker compose -f docker-compose.yml -f docker-compose.localstack.yml down

Cleaning Up Emulated AWS Resources #

Before stopping LocalStack, you may want to clean up specific AWS resources within the LocalStack environment:

# Set environment variables to point to LocalStack
export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# Delete S3 buckets
aws --endpoint-url=$AWS_ENDPOINT_URL s3 ls | awk '{print $3}' | xargs -I{} aws --endpoint-url=$AWS_ENDPOINT_URL s3 rb s3://{} --force

# Delete SQS queues
aws --endpoint-url=$AWS_ENDPOINT_URL sqs list-queues | jq -r '.QueueUrls[]' | xargs -I{} aws --endpoint-url=$AWS_ENDPOINT_URL sqs delete-queue --queue-url {}

# Delete Lambda functions
aws --endpoint-url=$AWS_ENDPOINT_URL lambda list-functions | jq -r '.Functions[].FunctionName' | xargs -I{} aws --endpoint-url=$AWS_ENDPOINT_URL lambda delete-function --function-name {}

# Delete DynamoDB tables
aws --endpoint-url=$AWS_ENDPOINT_URL dynamodb list-tables | jq -r '.TableNames[]' | xargs -I{} aws --endpoint-url=$AWS_ENDPOINT_URL dynamodb delete-table --table-name {}

Cleaning Up CDK Resources #

If you used AWS CDK with LocalStack:

# Navigate to your CDK directory
cd cdk

# Destroy all CDK stacks
npx cdk destroy --all --force --endpoint-url=http://localhost:4566

Removing Local Data #

Clean up any data directories and files created for LocalStack:

# Remove LocalStack data directory
rm -rf .localstack

# Remove CDK outputs and CloudFormation templates
rm -rf cdk/cdk.out

# Remove temporary deployment files
rm -f local-deploy.log

Lab-Specific Cleanup #

For the examples in our Lab8:

# Navigate to the lab directory
cd lab8_localStack_example

# Run the cleanup script
./cleanup.sh

# Or manually clean up
docker compose down -v
rm -rf .localstack
cd cdk && npm run clean && cd ..

Verifying Complete Cleanup #

Verify all resources have been properly cleaned up:

# Check if any Docker containers are still running
docker ps

# Check if any Docker volumes related to LocalStack remain
docker volume ls | grep localstack

# Check for remaining Docker networks
docker network ls | grep localstack

Resetting AWS CLI Configuration #

If you modified your AWS CLI configuration for LocalStack, you may want to reset it:

# Remove LocalStack profile if created
aws configure --profile localstack set aws_endpoint_url ""

# Or remove the profile entirely
aws configure --profile localstack set aws_access_key_id ""
aws configure --profile localstack set aws_secret_access_key ""
aws configure --profile localstack set region ""

Properly cleaning up your LocalStack environment ensures that you start with a fresh state for your next development session and prevents potential conflicts with future projects.

Conclusion #

LocalStack with Docker provides a powerful solution for local cloud development, enabling faster development cycles, cost savings, and improved workflow. By emulating AWS services locally, you can develop and test cloud applications without the need for constant deployment to actual cloud environments.

In this guide, we've explored:

  • The fundamentals of local cloud development
  • Setting up Docker with LocalStack
  • Creating a microservices application that works with emulated AWS services
  • Configuring the LocalStack environment
  • Deploying infrastructure using AWS CDK
  • Testing against emulated services
  • Improving development workflows
  • Managing common challenges and differences between local and production environments

By incorporating LocalStack into your Docker-based development workflow, you can create more efficient, cost-effective, and streamlined cloud development processes.

Have you used LocalStack in your Docker development workflow? Share your experiences or questions in the comments below!

Comments