Rathandev Chiluka's Blog

Simplify AWS RDS Log Downloads: A Step-by-Step Automation Guide

Rathandev Chiluka's photo
Rathandev Chiluka
·

4 min read

Table of contents

Introduction🌟

Fetching logs from AWS RDS is a part of day to day activity for DevOps/Cloud engineers. But downloading them by selecting and downloading one after the other may make the engineers frustrated. This script will make your job easy.

Script Overview📜

This script is written to retrieve Amazon RDS DB logs files within a specific time rage. It takes 3 arguments: DB instance identifier, log files start time, and end time respectively. The script uses the AWS CLI to list and download the logs files that fall within the specified time range.

Pre requisites✅

  • AWS IAM user/role used must have necessary permissions to describe and download RDS log files.

  • AWS cli has to be installed and configured properly.

  • jq has to be installed to parse AWS JSON outputs, extracting only relevant data like LastWritten time of log files

  • The shell script need to have execute permission.

  • The script requires 3 input parameters: DB instance identifier, log files start time, and end time respectively. For example ./rds-log-download demo-db '2025-01-25T16:30:00+05:30' '2025-01-26T06:30:00+05:30'.

Script Explanation🛠️

#!/bin/bash

#This script retrieves and downloads RDS DB logs within a specific time range.
#Example usage: ./rds-log-download demo-db '2025-01-25T16:30:00+05:30' '2025-01-26T06:30:00+05:30'

set -e
set -x

# Check if the correct number of arguments is provided
if [ "$#" -ne 3 ]; then
  echo "Error: Invalid number of arguments, refer below expected and example of parameters"
  echo "Expected parameters: $0 <DB_INSTANCE_IDENTIFIER> <Last written START_TIME> <Last written END_TIME>"
  echo "Example: $0 db-name '2025-02-05T16:30:00+05:30' '2025-02-05T06:30:00+05:30'"
  exit 1
fi

# Assign positional parameters to variables
DB_INSTANCE_IDENTIFIER=$1
START_TIME=$2
END_TIME=$3

# Convert START_TIME and END_TIME to Unix timestamps for comparison
start_time_unix=$(date -d "$START_TIME" +%s)000
end_time_unix=$(date -d "$END_TIME" +%s)000

# List all log files and store them in a temporary file
aws rds describe-db-log-files --db-instance-identifier $DB_INSTANCE_IDENTIFIER --output json > log_files.json

# Create a directory to store the downloaded logs
mkdir -p rds_logs

# Loop through each log file and download it if it matches the date and time range
jq -r '.DescribeDBLogFiles[] | select(.LastWritten >= '"$start_time_unix"' and .LastWritten <= '"$end_time_unix"').LogFileName' log_files.json | while read -r log_file; do
  echo "Downloading $log_file..."
  aws rds download-db-log-file-portion --db-instance-identifier $DB_INSTANCE_IDENTIFIER --log-file-name "$log_file" --output text > "rds_logs/$(basename $log_file)"
done

# Clean up of temporary file
rm log_files.json

echo "All relevant logs have been downloaded."

  • set -e: Ensures the script exits immediately if any command exits with a non-zero status. In linux if a command executes successfully then it exits with status code as zero, when a command exits with non-zero status means it has not executed successfully. To know the exit status of previous command that you’ve executed, just run echo $?.

  • set -x: Enables the debug mode, which means the executed commands will be printed in the terminal.

  • if [ "$#" -ne 3 ]; then

    echo "Error: Invalid number of arguments, refer below expected and example of parameters"

    echo "Expected parameters: $0 <DB_INSTANCE_IDENTIFIER> "

    echo "Example: $0 db-name '2025-02-05T16:30:00+05:30' '2025-02-05T06:30:00+05:30'"

    exit 1

    fi

    • Argument Check: Verifies that exactly 3 paramerters are provided. If not, it prints error message and exits.

      1. $#: Gives the no.of arguments passed to the script.

      2. $0: This gives the file name. For example, if we execute the script with

        ./rds-logs-retrieve.sh demo-db '2025-02-05T16:30:00+05:30' '2025-02-05T06:30:00+05:30'
        Here linux understands,
        rds-logs-retrieve.sh - $0
        ‘2025-02-05T16:30:00+05:30' - $1

        '2025-02-05T06:30:00+05:30' - $2

  • DB_INSTANCE_IDENTIFIER=$1

    START_TIME=$2

    END_TIME=$3
    Assign positional parameters to meaningful variables for better understanding.

  • start_time_unix=$(date -d "$START_TIME" +%s)000

    end_time_unix=$(date -d "$END_TIME" +%s)000

    Converts date and UTC timings to unix time stamps because AWS RDS often use Unix timestamps for logging and for ease of comparison.

  • aws rds describe-db-log-files --db-instance-identifier $DB_INSTANCE_IDENTIFIER --output json > log_files.json
    The above command describes the log files of mentioned DB and stores it in a temporary file called log_files.json(which will be auto deleted later).

  • mkdir -p rds_logs : I hope everyone knows what does this command do. For those who don’t know, it creates directory called rds_logs to store the log files.

  • jq -r '.DescribeDBLogFiles[] | select(.LastWritten >= '"$start_time_unix"' and .LastWritten <= '"$end_time_unix"').LogFileName' log_files.json | while read -r log_file; do echo "Downloading $log_file..." aws rds download-db-log-file-portion --db-instance-identifier $DB_INSTANCE_IDENTIFIER --log-file-name "$log_file" --output text > "rds_logs/$(basename $log_file)" done

    It will filter the log files of specified DB and time range data from log_files.json ( we’ve created in previous step) and this output is passed (with pipe |) to while loop to download the log files and stored in rds_logs directory.

  • rm log_files.json: Here it is a best practice to delete the unwanted files (log_files.json) to clear the space.

  • echo "All relevant logs have been downloaded." It prints the message between double codes on the terminal.

Conclusion🎉

Automating the download of AWS RDS log files can significantly enhance the efficiency and productivity of DevOps and Cloud engineers. By utilising the provided script, engineers can streamline the process, saving time and reducing frustration associated with manual log retrieval.

Thanks for reading patiently till the end. Feel free to comment if you stuck somewhere in this script or if you find more efficient ways to download the RDS log files.

AWSDevopsrdsautomationshell scriptbash script