The Case of the Disappearing ECS Agent (Or: How AWS Owes Me Five Days of My Life)

It started like any normal day. I was overhauling a CI/CD pipeline for a client, and I set up a clean AWS account for a sandbox environment. I'm working on a pretty solid framework for reusable cdk components, and I was deploying some containers to ECS, when suddenly everything went to hell. My tasks were stuck in a perpetual state of "PENDING" like they were waiting for a bus that would never arrive.

Mind you, I've done this before without any issues so what was going on?

🕵️‍♂️ The Mystery Begins

First, let me set the scene. I have this beautiful ECS cluster, running on EC2 instances (for this use case EC2 instances are more cost effective than Fargate). Everything was working fine until... it wasn't.

$ aws ecs list-tasks --cluster <my-cluster> --profile <my-profile>
{
    "taskArns": []
}

First things first, I did a remote session into the instance and checked the status of docker and the ecs agent.

sudo docker ps

Result:

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Ok, that's not good. The ECS agent is not running.

sudo systemctl status ecs

Result:

○ ecs.service - Amazon Elastic Container Service - container agent
     Loaded: loaded (/usr/lib/systemd/system/ecs.service; enabled; preset: disabled)
     Active: inactive (dead)
       Docs: https://aws.amazon.com/documentation/ecs/

🔍 The Investigation: Where's My Agent?

What the...? The ECS agent is not running and it's not even active.

Ok, let me recycle the instances and try again.

Kill the instances and let the auto scaling group recreate them

aws autoscaling terminate-instances-in-auto-scaling-group \
    --auto-scaling-group-name <my-auto-scaling-group> \
    --instance-ids <my-instance-id> \
    --profile <my-profile>

AND

aws autoscaling describe-auto-scaling-groups \
    --auto-scaling-group-name <my-auto-scaling-group> \
    --profile <my-profile>

Grab some coffee ☕️ while I wait for the instances to be terminated and the auto scaling group to recreate them. Actually it was pretty quick by the time I got back from grabbing another cup of coffee my instances were already up and running.

And then nothing. Same as before. My day just got a whole lot worse.

🐰 Down the Documentation Rabbit Hole

So I did what any self-respecting developer would do: I dove into the AWS documentation. Big mistake.

The AWS docs are like that friend who gives you directions but leaves out all the important turns. They'll tell you "check the agent logs" but won't tell you WHERE those logs actually are or what you're supposed to be looking for.

🕦 Time flies when you're investigating.

After hours of clicking through pages that looked like they were designed in 1998, and trying any and every suggestion I could find online, I was right back where I started. Without a clear plan to make this work.

Ok, let me try something else. I'll check the other working ECS cluster to see if I can find any clues. Nope they all looked like the same setup I just deployed (except they were working).

🔍 The Systematic Debugging Checklist

After methodically checking every component:

• Security Groups ✅
• Instance Role ✅
• ECS Cluster ✅
• Auto Scaling Group ✅
• Launch Template ✅
• AMI ✅
• Instance ✅
• ECS Agent ❌
• Docker ❌
• Panic Level ✅✅✅✅✅✅✅ 😱🙀

At this point, I'm questioning everything. Did I use the wrong AMI? Did AWS break something? Is this a cosmic sign that I should have become a baker instead of a developer?

🤯 The Plot Thickens

After digging through countless Stack Overflow posts and GitHub issues (most of which were answered by people saying "have you tried turning it off and on again?"), I discovered something horrifying.

The ECS-optimized AMI I was using? It wasn't actually optimized for ECS.

Now, this didn't make sense, my scripts are loading the optimized AMI ID from SSM Parameter Store. I checked the SSM Parameter Store and sure enough it was the correct AMI ID.

My instance was using the correct AMI ID, but the ECS agent was not running. I checked the instance details and sure enough it was using the correct AMI ID. What the...?

☕ The Coffee-Fueled Solution

Five days and approximately 47 cups of coffee later, I finally figured it out. Here's what happened:

The Problem

Here's where it gets really twisted. I THOUGHT I was using the official ECS Optimized AMI. But here's the kicker: you have to SUBSCRIBE to the ECS Optimized image in the AWS Marketplace before you can actually use it.

And what happens if you don't subscribe? Does AWS give you a nice error message? Does it tell you "hey, you need to subscribe to this AMI first"?

NO. Of course not.

Instead, AWS silently downgrades your Launch Template to use a plain Amazon Linux 2023 image. The worst part? The instance details still show the CORRECT ECS Optimized AMI ID, so you think you're using the right image when you're actually not.

It's like ordering a steak and getting a salad, but the receipt still says "steak." You're left wondering why you're still hungry while everyone around you is enjoying their perfectly cooked medium-rare.

The Fix 💪

You have two options here:

Option 1: Manual Installation (The Quick Fix)

Since you're on Amazon Linux 2023, you need to use dnf instead of yum (because apparently package managers change more often than I change my socks):

# Install the ECS agent
sudo dnf install -y amazon-ecs-init

# Enable and start the service
sudo systemctl enable ecs
sudo systemctl start ecs

# Configure it
sudo echo "ECS_CLUSTER=my-cluster" > /etc/ecs/ecs.config
sudo systemctl restart ecs

Option 2: Subscribe to the Official ECS Optimized AMI (The "Right" Way)

This is what I ultimately went with:

1. Go to the AWS Marketplace and search for "Amazon ECS-Optimized AMI"
2. Click "Subscribe" - you only need to do this ONCE per region (which is annoying, but whatever)
3. Accept the terms and wait for the subscription to activate
4. Recycle your instances hoping the Launch Template will now use the correct image

Pro tip: The Launch Template might not automatically pick up the change (because of course it doesn't). I had to make a minor change to my Launch Template in my deployment pipeline to force it to build a new one.

After that? Everything worked like magic. Tasks started running, containers deployed, and I could finally stop questioning my career choices.

Post Fix

sudo systemctl status ecs
● ecs.service - Amazon Elastic Container Service - container agent
     Loaded: loaded (/usr/lib/systemd/system/ecs.service; enabled; preset: disabled)
     Active: active (running) since Tue 2025-11-04 20:51:21 UTC; 5min ago
       Docs: https://aws.amazon.com/documentation/ecs/
    Process: 1897 ExecStartPre=/bin/bash -c if [ $(/usr/bin/systemctl is-active docker) != "active" ]; then exit 1; fi (code=exited, status=0/SUCCESS)
    Process: 1899 ExecStartPre=/usr/libexec/amazon-ecs-init pre-start (code=exited, status=0/SUCCESS)
   Main PID: 1934 (amazon-ecs-init)
      Tasks: 5 (limit: 4558)
     Memory: 131.5M
        CPU: 212ms
     CGroup: /system.slice/ecs.service
             └─1934 /usr/libexec/amazon-ecs-init start

Nov 04 20:51:19 ip-10-1-0-241.ec2.internal amazon-ecs-init[1899]: level=info time=2025-11-04T20:51:19Z msg="Successfully blocked IPv6 off-host access for introspection server with ip6tables."
Nov 04 20:51:19 ip-10-1-0-241.ec2.internal amazon-ecs-init[1899]: level=info time=2025-11-04T20:51:19Z msg="pre-start: checking ecs agent container image loaded presence"
Nov 04 20:51:19 ip-10-1-0-241.ec2.internal amazon-ecs-init[1899]: level=info time=2025-11-04T20:51:19Z msg="pre-start: ecs agent container image loaded presence: false"
Nov 04 20:51:19 ip-10-1-0-241.ec2.internal amazon-ecs-init[1899]: level=info time=2025-11-04T20:51:19Z msg="pre-start: reloading agent"
Nov 04 20:51:21 ip-10-1-0-241.ec2.internal systemd[1]: Started ecs.service - Amazon Elastic Container Service - container agent.
Nov 04 20:51:21 ip-10-1-0-241.ec2.internal amazon-ecs-init[1934]: level=info time=2025-11-04T20:51:21Z msg="Successfully created docker client with API version 1.25"
Nov 04 20:51:21 ip-10-1-0-241.ec2.internal amazon-ecs-init[1934]: level=info time=2025-11-04T20:51:21Z msg="start"
Nov 04 20:51:21 ip-10-1-0-241.ec2.internal amazon-ecs-init[1934]: level=info time=2025-11-04T20:51:21Z msg="No existing agent container to remove."
Nov 04 20:51:21 ip-10-1-0-241.ec2.internal amazon-ecs-init[1934]: level=info time=2025-11-04T20:51:21Z msg="Starting Amazon Elastic Container Service Agent"
Nov 04 20:51:21 ip-10-1-0-241.ec2.internal amazon-ecs-init[1934]: level=info time=2025-11-04T20:51:21Z msg="Operating system family is: amzn_2023"

🎭 The Emotional Rollercoaster

Let me describe the emotional journey:

• Day 1: Confusion. "This is weird, maybe it'll fix itself."
• Day 2: Frustration. "Why is nothing working? AWS hates me."
• Day 3: Despair. "I should have just used Kubernetes like everyone else."
• Day 4: Rage. "I'm going to write a strongly worded blog post about this!"
• Day 5: Triumph. "IT WORKS! I'M A GENIUS!"

💡 Lessons Learned (The Hard Way)

1. ALWAYS subscribe to the ECS Optimized AMI - Before you even think about using it, go to the AWS Marketplace and subscribe. Do this per region because AWS loves making you do things multiple times.
2. Verify your actual AMI content - Don't trust the AMI ID shown in the console. SSH in (or use Session Manager like a civilized person) and check if the ECS agent is actually installed.
3. Use dnf on AL2023 - yum is so 2022. Amazon Linux 2023 uses dnf, and yes, this matters.
4. Test your Launch Templates - Just because you subscribed doesn't mean existing templates will pick up the change. Force a rebuild if needed.

🛡️ Prevention: How to Avoid This Nightmare

Here's how you can prevent yourself from losing five days of your life:

1. Subscribe to the AMI First

# This can't be automated - you have to do it manually in the console
# Go to AWS Marketplace -> Search "Amazon ECS-Optimized AMI" -> Subscribe
# Remember: ONCE PER REGION because AWS hates convenience

2. Verify Your AMI is Actually ECS-Optimized

# Get the latest ECS-optimized AMI
aws ssm get-parameter --name /aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id

# Then SSH in and verify
sudo systemctl status ecs
# Note: /etc/ecs-release doesn't exist in AL2023 ECS Optimized AMIs
# The real check is whether the ECS service is active and running

3. Add Fallback to User Data

# Add this to your user data script as a safety net
sudo systemctl status ecs || {
    echo "ECS agent not found, installing manually..."
    sudo dnf install -y amazon-ecs-init  # Use dnf for AL2023!
    sudo systemctl enable ecs
    sudo systemctl start ecs
    echo "ECS_CLUSTER=my-cluster" > /etc/ecs/ecs.config
    sudo systemctl restart ecs
}

4. Monitor the Agent

Set up CloudWatch metrics to monitor the ECS agent. If it stops running, you'll know immediately instead of discovering it when your production deployment fails.

🎯 The Bottom Line

AWS is amazing until it's not. The ECS agent disappearing act wasn't about custom AMIs or missing services - it was about AWS silently downgrading my Launch Template because I hadn't subscribed to the marketplace AMI. This cost me five days of my life, countless cups of coffee, and what little sanity I had left.

The real kicker? The instance details showed the CORRECT AMI ID the entire time, making this one of the most misleading debugging experiences I've ever had.

But hey, at least I got a blog post out of it, right?

Dear AWS: If you're reading this, I expect compensation for those five days. I accept payment in AWS credits, coffee, or a heartfelt apology. And maybe fix the silent downgrade thing? That would be nice.

Dear Fellow Developers: Don't let this happen to you. Check your ECS agent. Check it often. And maybe keep some extra coffee on hand. Just in case.

---

Have you ever been fooled by AWS documentation or silent failures? Share your horror stories in the comments below. Misery loves company!