Query and Filter CloudWatch Logs with Insights

Intermediate10 minTrending

Use CloudWatch Logs Insights to search, filter, and analyze log data across log groups with powerful query syntax for rapid debugging.

Prerequisites

-AWS CLI v2 installed and configured
-Application writing logs to CloudWatch Logs

Steps

List available log groups

Browse available CloudWatch log groups to identify where your application or service writes its logs.

$ aws logs describe-log-groups --query 'logGroups[].logGroupName' --output table

Use --log-group-name-prefix to filter results, for example --log-group-name-prefix /aws/lambda/ to see only Lambda function logs.

Tail logs in real time

Stream live log events from a log group to your terminal for real-time debugging.

$ aws logs tail /aws/lambda/my-function --follow --since 5m --format short

The --since flag accepts relative times like 5m, 1h, or 1d as well as absolute ISO 8601 timestamps.

Start a Logs Insights query

Run a structured query to find error patterns in your logs over a specific time range.

$ aws logs start-query --log-group-name /aws/lambda/my-function --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50'

Logs Insights queries are asynchronous. The command returns a queryId that you use to fetch results in the next step.

Get query results

Retrieve the results of a previously started Logs Insights query using the query ID.

$ aws logs get-query-results --query-id 12345678-abcd-efgh-ijkl-example

Check the status field in the response. If it shows 'Running', wait a few seconds and retry. Results are only available when status is 'Complete'.

Analyze patterns with stats

Use the stats command to aggregate log data and identify trends like error rates or latency distributions.

$ aws logs start-query --log-group-name /aws/lambda/my-function --start-time $(date -d '24 hours ago' +%s) --end-time $(date +%s) --query-string 'filter @type = "REPORT" | stats avg(@duration), max(@duration), count(*) by bin(1h)'

Lambda REPORT lines contain execution duration, memory used, and billed duration. Use @type = REPORT to filter for these summary lines.

Full Script

full-script.sh

#!/bin/bash
# CloudWatch Logs Analysis
# Usage: ./analyze-logs.sh <log-group-name> [hours-back] [filter-pattern]

set -euo pipefail

LOG_GROUP="${1:?Usage: $0 <log-group-name> [hours-back] [filter-pattern]}"
HOURS_BACK="${2:-1}"
FILTER="${3:-ERROR}"

END_TIME=$(date +%s)
START_TIME=$((END_TIME - HOURS_BACK * 3600))

echo "=== Querying $LOG_GROUP for '$FILTER' (last ${HOURS_BACK}h) ==="

QUERY_ID=$(aws logs start-query \
  --log-group-name "$LOG_GROUP" \
  --start-time "$START_TIME" \
  --end-time "$END_TIME" \
  --query-string "fields @timestamp, @message | filter @message like /$FILTER/ | sort @timestamp desc | limit 100" \
  --output text --query 'queryId')

echo "Query ID: $QUERY_ID"
echo "Waiting for results..."

STATUS="Running"
while [ "$STATUS" = "Running" ] || [ "$STATUS" = "Scheduled" ]; do
  sleep 2
  RESULT=$(aws logs get-query-results --query-id "$QUERY_ID" --output json)
  STATUS=$(echo "$RESULT" | jq -r '.status')
done

echo "Status: $STATUS"
echo ""

MATCH_COUNT=$(echo "$RESULT" | jq '.results | length')
echo "Found $MATCH_COUNT matching log entries:"
echo ""

echo "$RESULT" | jq -r '.results[] | "\(.[] | select(.field == "@timestamp") | .value)  \(.[] | select(.field == "@message") | .value)"' | head -50

FAQ

Discussion

Loading comments...

#!/bin/bash # CloudWatch Logs Analysis # Usage: ./analyze-logs.sh <log-group-name> [hours-back] [filter-pattern] set -euo pipefail LOG_GROUP="${1:?Usage: $0 <log-group-name> [hours-back] [filter-pattern]}" HOURS_BACK="${2:-1}" FILTER="${3:-ERROR}" END_TIME=$(date +%s) START_TIME=$((END_TIME - HOURS_BACK * 3600)) echo "=== Querying $LOG_GROUP for '$FILTER' (last ${HOURS_BACK}h) ===" QUERY_ID=$(aws logs start-query \ --log-group-name "$LOG_GROUP" \ --start-time "$START_TIME" \ --end-time "$END_TIME" \ --query-string "fields @timestamp, @message | filter @message like /$FILTER/ | sort @timestamp desc | limit 100" \ --output text --query 'queryId') echo "Query ID: $QUERY_ID" echo "Waiting for results..." STATUS="Running" while [ "$STATUS" = "Running" ] || [ "$STATUS" = "Scheduled" ]; do sleep 2 RESULT=$(aws logs get-query-results --query-id "$QUERY_ID" --output json) STATUS=$(echo "$RESULT" | jq -r '.status') done echo "Status: $STATUS" echo "" MATCH_COUNT=$(echo "$RESULT" | jq '.results | length') echo "Found $MATCH_COUNT matching log entries:" echo "" echo "$RESULT" | jq -r '.results[] | "\(.[] | select(.field == "@timestamp") | .value) \(.[] | select(.field == "@message") | .value)"' | head -50