Pete Freitag Pete Freitag

Setup CloudWatch Logs Agent on Ubuntu 18.04 LTS

Published on August 06, 2019
By Pete Freitag
linuxweb

The AWS CloudWatch Logs Agent can be setup to push logs to the AWS CloudWatch Logs service from any server. In this howto, I'll show you how to set it up on Ubuntu 18.04 LTS, but you should be able to follow similar steps on Ubuntu 16.04 or other operating systems. The CloudWatch Logs agent can even be setup to collect EventViewer logs on Windows Servers.

What's also cool is that you can use this technique on any server, you don't have to be running in EC2, you can be using DigitalOcean, Linode, Google Cloud, Azure, etc. The AWS documentation doesn't really make this clear or give you much guidance for running on other providers (for obvious reasons), so here's a simple guide for you to follow.

You should also be able to easily bake this process into a configuration management script or add to a Dockerfile to use in your own Docker images.

Download / Install the Debian Package

AWS publishes the locations of the CloudWatch Logs Agent deb package in their documentation, we are just going to download it with curl and run it:

#download it
curl -o /root/amazon-cloudwatch-agent.deb https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb
#install it
dpkg -i -E /root/amazon-cloudwatch-agent.deb

All commands in this guide are assumed to be run as root or with sudo

Add cwagent User to adm group

Next we are going to modify the linux user account that the installer created cwagent and add it to the adm group, which will give it read permission to many of the default Ubuntu system logs.

usermod -aG adm cwagent

Setup an IAM User Account and Permissions

Now we need to give your Ubuntu server permission to publish its log data to your AWS account's CloudWatch Logs.

If you are running on AWS EC2, you can skip this step, and use the assumed role. You will still need to make sure that the assumed role has IAM permissions
Create an IAM user in the AWS Console and take note of the AWS accessID and AWS secretKey.

On your Ubuntu server create a file /home/cwagent/.aws/credentials with the following:

[AmazonCloudWatchAgent]
aws_access_key_id = aaaaaaaa
aws_secret_access_key = bbbbbbbb

Next create a file: /home/cwagent/.aws/config with the following:

[AmazonCloudWatchAgent]
output = text
region = us-east-1

Be sure to use the aws_access_key_id and aws_secret_access_key from the new IAM user you created in the AWS console. Also be sure that the region is specified correctly.

Next tell the CloudWatch Logs Agent to look here for credentials by appending to the /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml file:

echo "[credentials]" >> /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
echo '    shared_credential_file = "/home/cwagent/.aws/credentials"' >> /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

Create a Log Group

Take some time to think about how you want to organize your logs. CloudWatch Logs provides a notion of Log Groups and Log Streams, a log group can hold lots of log streams. Some like to create log groups dynamically, others may like all their server logs in the same group. In this tutorial we are going to create a single log group (eg web-server-log-group) that can hold logs for multiple servers by creating log streams like hostname.example.com/syslog

I like this approach because I can then limit the IAM permissions (setup in the next step) to only give access to publish logs and create streams in a single log group.

Grant the IAM User / Role Permission to Publish Logs

Assuming we have a log group named web-server-log-group we will need to attach an IAM policy to the IAM user you created in the previous step. Or if you are running on EC2 you can attach the policy to the assumed role. From the AWS console you can the Add inline policy button, and use something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchLogAgentPutForWebServerLogGroup",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:1234567890:log-group:web-server-log-group",
                "arn:aws:logs:us-east-1:1234567890:log-group:web-server-log-group:*:*"
            ]
        }
    ]
}

Note that the number 1234567890 should be replaced with your AWS account id.

If you want to allow the CloudWatch Logs Agent to publish metrics data (CPU, Memory, Network, Disk) then you will need to attach an additional policy to allow cloudwatch:PutMetricData

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchPutMetricData",
            "Effect": "Allow",
            "Action": "cloudwatch:PutMetricData",
            "Resource": "*"
        }
    ]
}

I need to do some more research to see if the Resource can be further constrained instead of using the wildcard *.

AWS also provides a builtin policy named CloudWatchAgentServerPolicy which you could use, but it is a bit over-permissive in my opinion. It will allow the server that the agent runs on to create new log groups, and publish logs to any log group.

Telling CloudWatch Logs Agent Which Log Files to Collect

The AWS CloudWatch Logs agent has a tool which you can run to create the json configuration file via a wizard:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Or you can just manually create a JSON file and place it here: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json - that file might look like this:

{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "cwagent",
		"credentials_path":"/home/cwagent/.aws/credentials"
	},
	"logs": {
		"logs_collected": {
			"files": {
				"collect_list": [
					{
						"file_path": "/var/log/syslog",
						"log_group_name": "foundeo-web",
						"log_stream_name": "{hostname}/syslog",
						"timestamp_format" :"%b %d %H:%M:%S"
					}
                                ]
                        }
               }
       }
}

The above will publish the contents of /var/log/syslog from your Ubuntu server to the AWS CloudWatch Logs service. Here is an example of sending nginx logs to CloudWatch Logs.

Enable and Start the CloudWatch Logs Agent service

Our final step is to enable the AWS CloudWatch Logs Ubuntu service (so that it will start after reboot), and then start the service, we can do that with these two commands:

systemctl enable amazon-cloudwatch-agent.service
service amazon-cloudwatch-agent start


aws cloudwatch logs ubuntu

Setup CloudWatch Logs Agent on Ubuntu 18.04 LTS was first published on August 06, 2019.

If you like reading about aws, cloudwatch, logs, or ubuntu then you might also like:

Weekly Security Advisories Email

Advisory Week is a new weekly email containing security advisories published by major software vendors (Adobe, Apple, Microsoft, etc).

Comments

I'm having a difficult time installing the agent. There seem to be 10 different ways to use cloudwatch with a linux system. Thank you for a blog that actually matches the documentation for the most part.

I'm confused where you got this part:

"credentials_path":"/home/cwagent/.aws/credentials"

That doesn't seem to be an accepted field: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html

This blog is the only other blog I found that follows official documentation methods https://docs.bitnami.com/aws/faq/administration/install-use-cloudwatch/ and their method doesn't work either (modifying common-config.toml)

I'm getting this error in the logs

refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed
by Nicholas Brady on 10/28/2019 at 11:11:10 PM UTC
Hello, How can i send syslog logs from Ubuntu to Cloudwatch with out instances?
by Cesar on 02/07/2020 at 3:06:06 PM UTC
Hey Pete, great writeup!

One suggestion to improve security -- create an IAM Role that leverages a managed IAM Policy that has access to CloudWatch logs. Then apply the role to your instance instead of storing credentials on it.

This way if your instance is ever compromised, you don't have to worry about the attacker gaining access to your AWS key. A lot of damage could be done before you have a chance to revoke it.

Btw, I used to attend CF.Objective in a former life and I loved your sessions. Glad to see you're still keeping the CF evangelism alive!
by Jake Hand on 08/13/2020 at 5:03:09 PM UTC
@Jake - yes, I briefly mentioned that, but since the main goal here was setting this up outside of EC2 I wanted to show how to use credentials.

Wow cf.Objective() -- blast from the past!
by Pete Freitag on 09/10/2020 at 7:35:17 PM UTC
So to make it work on EC2, in the agent part of the config file i put
"agent":{
"metrics_collection_interval": 60,
"run_as_user": "root"
}
by Colin on 01/27/2021 at 5:14:06 PM UTC
Also, change the log_group_name to the group name created earlier
by Colin on 01/27/2021 at 5:14:58 PM UTC