ECS Service Rolling Deployment and Troubleshooting

Advanced20 min

Perform zero-downtime rolling deployments of ECS Fargate services, monitor task health, and troubleshoot failed deployments with circuit breaker rollbacks.

Prerequisites

-AWS CLI v2 installed and configured
-Existing ECS cluster and service
-Container image pushed to ECR

Steps

Register a new task definition revision

Create a new task definition revision with the updated container image tag to prepare for deployment.

$ aws ecs describe-task-definition --task-definition my-app --query taskDefinition > task-def.json && jq '.containerDefinitions[0].image = "111111111111.dkr.ecr.us-east-1.amazonaws.com/my-app:v2.1.0"' task-def.json > updated-task-def.json && aws ecs register-task-definition --cli-input-json file://updated-task-def.json

Only update the image tag, not the entire task definition. This preserves CPU, memory, environment variables, and other settings.

Remove taskDefinitionArn, revision, status, registeredAt, registeredBy, and requiresAttributes from the JSON before re-registering, or the API will reject the request.

Update the service to trigger deployment

Update the ECS service to use the new task definition revision, triggering a rolling deployment.

$ aws ecs update-service --cluster my-cluster --service my-app-service --task-definition my-app:5 --force-new-deployment

The --force-new-deployment flag ensures tasks are replaced even if the task definition revision has not changed, useful for pulling the latest image with a mutable tag.

Monitor the deployment progress

Watch the service deployments to track the rollout of new tasks and draining of old tasks.

$ aws ecs describe-services --cluster my-cluster --services my-app-service --query 'services[0].deployments[*].{Status:status,Running:runningCount,Desired:desiredCount,Pending:pendingCount,Revision:taskDefinition}' --output table

Check task health and stopped reasons

List recently stopped tasks and inspect their stop reasons to identify why tasks might be failing during deployment.

$ aws ecs list-tasks --cluster my-cluster --service-name my-app-service --desired-status STOPPED --query 'taskArns[0:5]' --output text | xargs -I {} aws ecs describe-tasks --cluster my-cluster --tasks {} --query 'tasks[*].{TaskId:taskArn,StopCode:stopCode,Reason:stoppedReason,Container:containers[0].reason}' --output table

ECS only retains stopped task information for about 1 hour. Check stopped tasks promptly during troubleshooting.

View container logs for failed tasks

Inspect CloudWatch logs for containers that failed health checks or crashed during startup.

$ aws logs tail /ecs/my-app --since 10m --format short | head -100

If the container crashes immediately, check for missing environment variables, invalid entrypoint commands, or insufficient memory allocation in the task definition.

Roll back if deployment fails

If the circuit breaker has not auto-rolled back, manually revert to the previous task definition revision.

$ aws ecs update-service --cluster my-cluster --service my-app-service --task-definition my-app:4 --force-new-deployment && aws ecs wait services-stable --cluster my-cluster --services my-app-service

Enable the deployment circuit breaker with rollback in your service configuration to automatically revert failed deployments.

Full Script

full-script.sh

#!/bin/bash
# ECS Service Rolling Deployment
# Usage: ./deploy-ecs.sh <cluster> <service> <image-uri>

set -euo pipefail

CLUSTER="${1:?Usage: $0 <cluster> <service> <image-uri>}"
SERVICE="${2:?Provide service name}"
IMAGE_URI="${3:?Provide full image URI with tag}"

echo "=== Getting current task definition ==="
TASK_DEF_ARN=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
  --query 'services[0].taskDefinition' --output text)
TASK_DEF_FAMILY=$(echo "$TASK_DEF_ARN" | awk -F'/' '{print $NF}' | awk -F':' '{print $1}')

echo "Current task definition: $TASK_DEF_ARN"

echo "=== Creating new task definition revision ==="
TASK_DEF=$(aws ecs describe-task-definition --task-definition "$TASK_DEF_ARN" --query taskDefinition)

NEW_TASK_DEF=$(echo "$TASK_DEF" | jq \
  --arg image "$IMAGE_URI" \
  'del(.taskDefinitionArn, .revision, .status, .registeredAt, .registeredBy, .requiresAttributes, .compatibilities) |
   .containerDefinitions[0].image = $image')

NEW_REVISION=$(echo "$NEW_TASK_DEF" | aws ecs register-task-definition --cli-input-json file:///dev/stdin \
  --query 'taskDefinition.taskDefinitionArn' --output text)

echo "New revision: $NEW_REVISION"

echo "=== Updating service ==="
aws ecs update-service \
  --cluster "$CLUSTER" \
  --service "$SERVICE" \
  --task-definition "$NEW_REVISION" \
  --force-new-deployment \
  --query 'service.deployments[*].{Status:status,Running:runningCount,Desired:desiredCount}' \
  --output table

echo "=== Waiting for service to stabilize ==="
echo "This may take several minutes..."
if aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE"; then
  echo "Deployment successful!"
  aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
    --query 'services[0].deployments[*].{Status:status,Running:runningCount,TaskDef:taskDefinition}' \
    --output table
else
  echo "Deployment did not stabilize. Checking stopped tasks..."
  aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" --desired-status STOPPED \
    --query 'taskArns[0:3]' --output text | tr '\t' '\n' | while read -r TASK; do
      echo "---"
      aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK" \
        --query 'tasks[0].{StopCode:stopCode,Reason:stoppedReason}' --output table
    done
  exit 1
fi

FAQ

Discussion

Loading comments...

#!/bin/bash # ECS Service Rolling Deployment # Usage: ./deploy-ecs.sh <cluster> <service> <image-uri> set -euo pipefail CLUSTER="${1:?Usage: $0 <cluster> <service> <image-uri>}" SERVICE="${2:?Provide service name}" IMAGE_URI="${3:?Provide full image URI with tag}" echo "=== Getting current task definition ===" TASK_DEF_ARN=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \ --query 'services[0].taskDefinition' --output text) TASK_DEF_FAMILY=$(echo "$TASK_DEF_ARN" | awk -F'/' '{print $NF}' | awk -F':' '{print $1}') echo "Current task definition: $TASK_DEF_ARN" echo "=== Creating new task definition revision ===" TASK_DEF=$(aws ecs describe-task-definition --task-definition "$TASK_DEF_ARN" --query taskDefinition) NEW_TASK_DEF=$(echo "$TASK_DEF" | jq \ --arg image "$IMAGE_URI" \ 'del(.taskDefinitionArn, .revision, .status, .registeredAt, .registeredBy, .requiresAttributes, .compatibilities) | .containerDefinitions[0].image = $image') NEW_REVISION=$(echo "$NEW_TASK_DEF" | aws ecs register-task-definition --cli-input-json file:///dev/stdin \ --query 'taskDefinition.taskDefinitionArn' --output text) echo "New revision: $NEW_REVISION" echo "=== Updating service ===" aws ecs update-service \ --cluster "$CLUSTER" \ --service "$SERVICE" \ --task-definition "$NEW_REVISION" \ --force-new-deployment \ --query 'service.deployments[*].{Status:status,Running:runningCount,Desired:desiredCount}' \ --output table echo "=== Waiting for service to stabilize ===" echo "This may take several minutes..." if aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE"; then echo "Deployment successful!" aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \ --query 'services[0].deployments[*].{Status:status,Running:runningCount,TaskDef:taskDefinition}' \ --output table else echo "Deployment did not stabilize. Checking stopped tasks..." aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" --desired-status STOPPED \ --query 'taskArns[0:3]' --output text | tr '\t' '\n' | while read -r TASK; do echo "---" aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK" \ --query 'tasks[0].{StopCode:stopCode,Reason:stoppedReason}' --output table done exit 1 fi