Add CloudWatch logs to your Docker Container

Take control of your application logs with CloudWatch!

In this post, I'm going to show you how easy it is to wire up a docker container to log to CloudWatch. 

In this example, I'm going to assume the following:

1. You are using an EC2 instance
2. You have docker installed on the EC2 instance and you're running a container

The next two are items are requirements as well but we'll cover them here:

1. You have an EC2 instance with the CloudWatch agent running.
2. You have the correct permissions via a role on the EC2 instance.

The CloudWatch Agent

Use the following scripts to install the CloudWatch agent.  I typically include this in my user-data scripts to automatically install on the EC2 instance

# get the package
sudo wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

# install the agent
sudo rpm -U ./amazon-cloudwatch-agent.rpm

# start it
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a start

# make sure it's running
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

The AWS IAM Permissions

In order for your EC2 instance to submit logs through the CloudWatch agent, the agent must have permission to do so. The easiest way to accomplish this is to add them to a Role that the EC2 instance has assigned to it.

The minimum permissions are the following:

An example of the policy would look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}

AWS CDK

If you are using the AWS CDK it would look something like this:

using Amazon.CDK.AWS.IAM;
...

private PolicyStatement GenerateWriteAccess()
{
   var statementProps = new PolicyStatementProps
   {
      Effect = Effect.ALLOW,
      Actions = new string[] {
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:CreateLogGroup"
      },
     Resources = new string[] { "*" }
   };

  var statement = new PolicyStatement(statementProps);

  return statement;
}

Docker Compose

Finally, Add the logging options to your docker-compose file:

logging:
  driver: "awslogs" 
  options:
    awslogs-stream: "THE_STREAM_NAME"
    awslogs-group: "THE_LOG_GROUP_NAME"
    awslogs-region: "THE_AWS_REGION"
    awslogs-create-group: "true"

THE_STREAM_NAME is the name of the logging stream. Like a file name. Mine are typically in the form of the docker-service name and the EC2 instance Id, like web-api-service-instance-id.

THE_LOG_GROUP_NAME is the name that you see under the CloudWatch Log groups section and have /'s in them to help differentiate them.  Mine are typically in the form of env/app-name like:

• dev/company/app-name
• prod/company-name/app-name

THE_AWS_REGION is the AWS region you are logging them to e.g. us-east-1

I typically have a shell script that dynamically creates my docker-compose file based on environment variables. The variables are loaded and written to the file similar to the script below.

You'll see in some sections, I'm actually embedding or swapping out fillers for the environment variables (see the __VARS__ and sed command).  I'm doing this out of convenience to make sure the compose file doesn't have any problems but you could use the environment variables directly - assuming you make sure they are always set.

If all goes well, you should have something in your cloud logs that looks something like the following

Side note: the cloudwatch-agent and user-data logs streams shown above were not covered by this post.  Those were configured through the amazon-cloudwatch-agent.json configuration settings of the CloudWatch agent.