Getting Started with Collecting and Managing AWS Logs

Logging is a critical component of any cloud-based infrastructure, and AWS offers a wide range of services for logging, monitoring, and analyzing logs such as CloudWatch Logs, CloudTrail, and Elasticsearch. These services allow you to collect and store log data, set up alerts, and perform advanced analysis on your logs. By using these services, you can gain visibility into the health and performance of your infrastructure, troubleshoot issues, and comply with regulatory requirements.

In this article, we will go over the basics of logging on AWS, including setting up log collection, sending logs to different destinations, and creating alerts. We will also provide examples of how to use these services to solve common logging challenges. Whether you're new to AWS or an experienced user, this guide will help you get started with logging on AWS and make the most of the services provided by AWS.

What logs do AWS generate?

Before we proceed with the rest of this article, let's briefly discuss how logs are generated in AWS and some of the commonly generated logs that you are likely to encounter when using the AWS platform.

There are two primary log sources: the AWS services, and the applications running on such services. Amazon's built-in log management service, CloudWatch, is the primary tool for collecting and aggregating such logs, but there's also CloudTrail which stores events describing all user and API activity.

Once the log data is aggregated, it can be monitored and analyzed within the tools or archived in AWS S3. It can also be forwarded to a different log management solution such as Logtail.

AWS Lambda logs

AWS Lambda is an event-driven compute service that lets you execute business logic in response to a wide variety of triggers without provisioning or managing servers. Several logs are generated by the service whenever a function is executed, and these logs provide various details about the function execution such as the start and end time, any error messages, and various metrics that can help you optimize performance and alert you to application-level issues.

Lambda logs are automatically sent to CloudWatch Logs in real-time, and can be viewed using the CloudWatch Logs console, the CloudWatch Logs API, or the AWS CLI. There are three types of logs that are generated by AWS Lambda:

1. Function logs: these are messages emitted to the standard output and standard error from a Lambda function. Each Lambda function will output its logs to separate log groups (/aws/lambda/<FunctionName>) and streams (YYY/MM/DD/[<FunctionVersion>]<InstanceId>). Ensure to use a logging framework to classify your log messages appropriately (through log levels) and output them in JSON format so that they can be queried, filtered and exported easily in CloudWatch Logs.

2. Extension logs: Lambda extensions are used to integrate the Lambda execution environment with various tools for observability, monitoring, security, and more. Log output from such extensions are streamed to CloudWatch so that you can analyze them to identify extension-related problems.

3. Platform logs: these logs are generated by the Lambda execution environment and they record events and errors related to function invocations and extensions. Such events include start and end time for an invocation, various metrics about the invocation (such as the duration of the function's execution and the memory usage), the request ID that uniquely identifies the function execution, and more.

   {
    "time": "2020-08-20T12:31:32.123Z",
    "type": "platform.report",
    "record": {"requestId": "6f7f0961f83442118a7af6fe80b88d56",
        "metrics": {"durationMs": 101.51,
            "billedDurationMs": 300,
            "memorySizeMB": 512,
            "maxMemoryUsedMB": 33,
            "initDurationMs": 116.67
        }
    }
   }

AWS API Gateway logs

AWS API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. It generates logs that provide detailed information about the requests and responses processed by the API Gateway which are categorized into two:

1. Access logs

They are similar to Apache or NGINX access logs as they contain details about each request that passes through the API Gateway. Such logs provide a summary of the request by including details such as the time the request occurred, the HTTP status code, the resource that was requested, and more.

   {
    "requestId": "e6d3cd70-655b-4e07-8405-d257b768a90b",
    "ip": "102.89.43.101",
    "caller": "-",
    "user": "-",
    "requestTime": "18/Jan/2023:12:40:20 +0000",
    "httpMethod": "GET",
    "resourcePath": "/",
    "status": "200",
    "protocol": "HTTP/1.1",
    "responseLength": "1310"
   }

2. Execution logs

These document all the processes happening within the API Gateway for each request. They contain a lot more detail about each request compared to access logs so they should typically only be enabled during troubleshooting sessions to avoid incurring heavy CloudWatch costs. For example, the logs in the screenshot below are for a single request.

Amazon S3 logs

Amazon S3 (Simple Storage Service) is a fully managed service that enables you to store, retrieve and manage data in the cloud. It stores data as objects (files and their metadata) within buckets (containers for objects), and gives each object a unique indentifier.

S3 access logs help you keep track of how each object in your various buckets are accessed and used for auditing or compliance purposes. They are stored in a separate S3 bucket and they include information such as the requester ID, the request type (e.g. GET, PUT, DELETE), the request date and time, the bucket name, object size, and more.

Here's what an S3 access log looks like:

79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be DOC-EXAMPLE-BUCKET1 [06/Feb/2019:00:00:38 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be 3E57427F3EXAMPLE REST.GET.VERSIONING - "GET /DOC-EXAMPLE-BUCKET1?versioning HTTP/1.1" 200 - 113 - 7 - "-" "S3Console/0.4" - s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader DOC-EXAMPLE-BUCKET1.s3.us-west-1.amazonaws.com TLSV1.2 arn:aws:s3:us-west-1:123456789012:accesspoint/example-AP Yes

You'll need to enable S3 server access logging for the bucket you'd like to track and specify the target bucket (where the logs will be stored) and prefix for the logs. Once enabled, S3 will automatically deliver the logs to the specified bucket.

Amazon RDS logs

Amazon RDS (Relational Database Service) is a fully managed service provided by AWS that makes it easy to set up, operate, and scale a relational database in the cloud. It supports various database engines such as MySQL, MariaDB, Oracle, SQL Server, and PostgreSQL.

Amazon RDS generates a variety of logs to help you monitor and troubleshoot your database instances. These logs include:

1. Error Logs: containing information about errors and warning messages that occur when running your database.

2. Slow Query Logs: indicates slow-performing queries that take longer than a specified time to execute.

3. Audit Logs: for tracking database activities such as logins, user activity, and database changes.

4. General Logs: contains information about all client connections and disconnections, as well as SQL statements that are executed by the server.

5. Binary Logs: records all changes made to the database, such as data modification statements and table structure changes.

It's worth noting that for some database engines, the log types that are available and how to access them may be different. You can view the logs in the RDS console, download them to your local computer, or stream them to CloudWatch Logs for further analysis and storage.

AWS CloudTrail logs

AWS CloudTrail is a service that allows you to record and track activity in your AWS account. It records all AWS Management Console sign-in events and API calls made in your account so that such data may be used to monitor user activity, troubleshoot issues, and audit the use of your AWS resources.

The logs generated by CloudTrail are stored in S3 bucket, and you can access them using the S3 console, the AWS CLI, or the S3 API. CloudTrail logs include the following information such as:

• The identity of the user or role that made the request.
• The time of the request.
• The source IP address of the request.
• The request parameters.
• The response elements returned by the service.

CloudTrail logs can be used for various use cases such as compliance, security, operational troubleshooting, and incident response. You can also use CloudTrail logs in conjunction with other AWS services such as Amazon CloudWatch, Amazon Elasticsearch Service, and AWS Lambda to create custom monitoring and automation solutions.

Collecting and viewing AWS logs in CloudWatch

Before you can derive value from your AWS logs, you need to collect them first and centralize them in one place. CloudWatch is the primary logging and monitoring service for the AWS platform, and it can help with collecting and centralizing logs and metrics from various AWS offerings such as the ones discussed in the previous section. It also provides search and analysis functionality to help you derive value from your log data, and it can also alert you to anomalies or other patterns in your logs.

Here are the steps to collect logs from AWS services using CloudWatch Logs:

1. Create a log group: A log group is a container for your logs. It is a collection of log streams that share the same log retention policy, access control, and monitoring settings. Generally, you'll create a different log group for each service or application that you want to monitor.

2. Create a log stream: A log stream is a sequence of log events that come from the same source. Each distinct source of logs in CloudWatch Logs is represented by a separate log stream. There can be an unlimited number of streams in each log group.

3. Enable logging for the service or application: You can enable logging for most AWS services through the AWS Management Console or by using the AWS CLI or SDKs.

4. Send the service/application logs to CloudWatch: Once logging is enabled for a service or application, ensure that the logs are being transmitted to CloudWatch Logs. In many cases, the functionality to automatically send logs to CloudWatch is already built-into the service. However, if such functionality is not available, you can also employ the CloudWatch agent to collect metrics and logs from EC2 insances and on-premise servers, or use the AWS CLI or API as appropriate.

5. View your log data: The CloudWatch Logs console, API, or the AWS CLI may be used to view and analyze the collected log data. If you're using the web console, you need to select the log group and log stream that you want to view and then inspect the individual log events in the stream.

Searching and querying your log data

CloudWatch Logs provides several ways to search, query, and filter log data in order to find the specific information you need. For example, the CloudWatch console provides a basic way to search your log data using filter patterns. Once you're on the Log events page, you can enter a filter pattern to search for and match terms, phrases, or values in your log events.

CloudWatch Logs Insights is another powerful tool that offers a query language that allows you to filter, aggregate, and perform calculations on your log data. You can use the CloudWatch Logs Insights console or the CloudWatch Logs API to run queries and visualize the results.

Creating metrics from log data

Another useful way to use log data stored in CloudWatch logs is by turning them into numerical CloudWatch metrics that you can visualize using dashboards or monitor and respond to them via CloudWatch alarms. For example, you can monitor your error rate, 4xx rate, the occurrences of a specific term in your logs, and more.

Visualizing AWS log data through CloudWatch dashboards

CloudWatch Dashboards is a feature provided in CloudWatch that allows you to view the performance and health of your AWS resources and applications in a single place, which can help you quickly identify trends and troubleshoot issues.

With Dashboards, you can create custom visualizations to display CloudWatch metrics, alarms, and logs. You can do this through the AWS Management Console, the AWS CLI, or the CloudWatch Dashboards API. Once a dashboard is created, you can add one or more widgets to it, and each one can display a specific metric, alarm, or log group. Widget can also be customized with different visualizations, such as line, stacked area, and bar charts, pie charts, and more.

CloudWatch Dashboards also support CloudWatch Logs Insights which allows you to have a dynamic view of your logs by running ad-hoc queries and viewing the results in widget form. You can also use dashboards to communicate the status of your services with other stakeholders by creating a URL or embedding the dashboard in your application or website using an iframe.

Exporting CloudWatch logs to Amazon S3

By default, CloudWatch Logs retains logs indefinitely, meaning that they will not be automatically deleted. However, as your services grow, this behavior can incur prohibitive costs and make searching through them more challenging. You can configure a log retention period for individual log groups in CloudWatch Logs so that logs older than that period will be deleted automatically.

If you want to retain your older logs for a longer period without incurring heavy costs, exporting them Amazon S3 is a common way to archive them. It also allows you to use S3's lifecycle policies to automatically move the logs to other storage classes or even other services such as Amazon Glacier or Amazon S3-IA.

Once you've configured CloudWatch to archive your logs in S3, you'll be able to access them and use the S3's features such as versioning, access management, and data lifecycle policies to store, protect and analyze your logs.

Exporting CloudWatch logs to other destinations

Besides archiving your AWS logs in S3, you can also send them to a third-party platform. This is a way to gain advanced visibility and analysis capabilities for your logs, and also allows you to use your preferred log analysis tool. For example, you can send your CloudWatch logs to Logtail and benefit from Logtail's more modern interface that provides a more cost-effective log management solution for your serverless logs and correlate them with the rest of your telemetry data.

To achieve this, you can a CloudWatch Logs subscription filter for the specific log group and choose the Lambda subscription option which allows you to automatically send log events to an AWS Lambda function for further processing and analysis.

You'll need to create a Lambda function that can process the log events from CloudWatch Logs, and this function should include the necessary logic to parse, filter, and send the log events to the desired location. With Logtail, you can use the HTTP REST API in your Lambda function to route a single event or a list of events to the service.

See the AWS CloudWatch documentation for more details on how to access CloudWatch logs in AWS Lambda.

Best practices for managing AWS log data

Monitoring logs in AWS is an important part of maintaining the security, performance, and availability of your resources and applications. Here are a few best practices for collecting and managing your AWS log data:

• Collect and store logs from all of your AWS services and resources, including EC2 instances, Lambda functions, and CloudTrail. Use CloudWatch Logs or other services like S3, Elasticsearch, or Kinesis Data Streams to store and aggregate your logs.

• Use CloudWatch Logs Insights to search, query, and analyze your logs in real-time. This will allow you to quickly identify issues and troubleshoot problems.

• Create CloudWatch Alarms to automatically notify you when specific conditions are met in your logs. You can also use SNS to send email or SMS notifications, or invoke a Lambda function to take automated actions.

• Visualize your metrics, alarms, and logs in a single place through CloudWatch Dashboards. This allows you to quickly view the performance and health of your resources and applications, and share this information with others.

• Use third-party log analysis tools like Logtail to gain advanced visibility and analysis capabilities for your logs.

• Implement security and compliance best practices for your logs such as encryption.

• Regularly review your logging strategy and archive your logs, keeping only the logs that you need for a certain period of time and delete the rest, this way you can save on storage and processing cost and comply with the legal and regulatory requirements.

Final thoughts

In this article, we aimed to provide you with the basics of logging on AWS and show you how to get started with logging on AWS. We hope this guide has been helpful in understanding the different logging services provided by AWS and how to use them effectively. With the right logging strategy in place, you can ensure that your infrastructure is running smoothly and that you have the information you need to make informed decisions.

Thanks for reading, and happy logging!

Article by

Ayooluwa Isaiah

Ayo is the Head of Content at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he’s not writing or coding, he loves to travel, bike, and play tennis.

Got an article suggestion? Let us know

Next article

The Missing Guide to AWS Lambda Logs

Discover how to manage AWS Lambda logs effectively: interpret auto-generated logs, structure for easy troubleshooting, and use Better Stack for cost-efficient monitoring

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.