AWS CloudWatch is a monitoring tool for the AWS platform. CloudWatch logs are an important resource for monitoring and helping to interpret the data in your AWS cloud.
This article covers the essentials of working with AWS CloudWatch and CloudWatch Logs.
What Can You Do with CloudWatch?
As a tool, CloudWatch is quite versatile. IT Pros can use it for several different purposes, including tracking performance metrics, setting threshold alarms, and even taking automated action when a monitored resource exceeds a predetermined threshold.
Monitor Amazon EC2 Instances
One of the most common uses of AWS CloudWatch is for the monitoring of EC2 instances. The nice thing about this functionality is that it is enabled by default. AWS collects performance metrics from EC2 instances every five minutes, and stores those metrics for 15 months so that you can monitor performance changes over time.
For instances that require more timely performance data, AWS does provide an option to collect performance data every minute. Doing so requires you to enable detailed monitoring for the instance, which is a simple process, but incurs an additional cost.
Monitor Events Logged by CloudTrail
AWS CloudWatch logs can do far more than simply monitor the performance of EC2 instances. You can also use CloudWatch to gather the events that have been monitored by AWS CloudTrail. For those who might not be familiar with CloudTrail, it is designed to be an auditing mechanism for AWS.
As you are no doubt aware, AWS is made up of an extremely diverse collection of services. The one thing that all of these services have in common is that they are built around the use of APIs. Any time that you interact with an AWS service, an API is at work in the background. This holds true regardless of whether the service is accessed programmatically, through the AWS console, or through the AWS CLI. CloudTrail’s job is to capture a record of all API activity that occurs across an AWS account. A log of the activity is written to an S3 bucket, but it is also possible to deliver the logging data to CloudWatch.
Kinesis Streams and AWS Lambda
AWS Kinesis Streams are designed to help AWS subscribers to either process or analyze extremely high volumes of streaming data. A Kinesis stream can capture data from hundreds of thousands of sources simultaneously, and can process or analyze multiple terabytes of data every hour. Kinesis is often used in conjunction with AWS Lambda, which allows for the automatic processing of streaming data. Lambda is designed to log data through CloudWatch logs.
Filtering and Searching AWS CloudWatch Logs
AWS CloudWatch logs can accumulate vast amounts of data, so it is important to be able to filter the log data based on your needs. Filtering is achieved through the use of metric filters. Perhaps the most important thing to understand about metric filters is that they do not support retroactive filtering. Only events that have been logged since the time that the filter was created will be reported in the filtered results. Log entries that existed prior to the filter’s creation are not included in the filtered results.
Creating a Metric Filter
To create a metric filter, log into the AWS console, and choose the CloudWatch service. When the CloudWatch dashboard appears, click on the Logs option, and then click on the number of metric filters that is displayed within your log group. (The number of metric filters will initially be set at zero.) If no log groups exist, you will have to create a log group before continuing.
Click the Add Metric Filter button, and you will be taken to a screen that asks you to specify a few different pieces of information. First, you will need to provide a filter pattern. A filter pattern specifies what the metric filter will look for within the log. (For instance, entering the word Error will cause the filter to look for occurrences of the word Error.)
Next, you will need to select the log data that you plan to test. Once you have made your selection, click the Test Pattern button to make sure that the results are what you expect, and then click on the Assign Metric button.
The resulting screen requires you to enter a filter name. The filter name is just a friendly name used to identify the metric filter within the log group. You will also need to specify a metric namespace. A metric namespace is nothing more than a group that contains related metrics. By default, AWS uses LogMetrics as the metric namespace name.
Finally, you will have to specify a metric name. The metric name is the name of the CloudWatch metric where the log information will be published. AWS also gives you the optional ability to write a metric value to the log when a pattern match occurs.
When you are done, click the Create Filter button, and the metric filter will be created. You can monitor your metrics from the CloudWatch Metrics dashboard.
