AWS CloudWatch logs is an useful logging system, but it has two quircks. It does not allow you too set a default retention period for newly created log groups, and it does not delete empty log streams that are older than the retention period. This utility:
You can use it as a command line utility. You can also install it as an AWS Lambda function and have your logs kept in order, NoOps style!
to install the log minder, type:
pip install aws-cloudwatch-log-minder
to set the default retention period on log groups without one, type:
cwlog-minder --dry-run set-log-retention --days 30
This will show you which log groups will have its retention period set. Remove the
it the retention period will be set. If you wish to set the retention of all log groups to the same
cwlog-minder --dry-run set-log-retention --days 30 --overwrite
To delete empty log streams older than the retention period, type:
cwlog-minder --dry-run delete-empty-log-streams
This will show you which empty log streams will be deleted. Remove the
these stream will be deleted.
To deploy the log minder as an AWS Lambda, type:
git clone https://github.com/binxio/aws-cloudwatch-log-minder.git
aws cloudformation deploy \
--capabilities CAPABILITY_IAM \
--stack-name aws-cloudwatch-log-minder \
--template-file ./cloudformation/aws-cloudwatch-log-minder.yaml \
This will install the log minder in your AWS account and run every hour.
To delete empty log groups, type:
cwlog-minder --dry-run delete-empty-log-groups
This will show you which empty log groups will be deleted. Remove the
these groups will be deleted. Do not use this command, if your log groups are
managed by CloudFormation or Terraform.
AWS regions and credential profiles can be selected via command line arguments or environment variables.
cwlog-minder --region eu-west-1 ...
cwlog-minder --profile dev ...
I was running log minder on a big collection of log steams. Since there was no log output I thought the tool was stuck or not working. When I turned on Debug logging I discovered it was doing a lot of log deletion. No log messages were shown because they were set to debug level.
Maybe a summary log message every 10 seconds can help show progress so that users don't think the script is not working.
I have been running an application for more than 3 years without the log minder. There are >100k log streams in 1 log group. With the rate limits of AWS Cloudwatch and the default maximum log minder duration of 5 minutes it would take weeks to clean up all the log streams.
For now I've been running log-minder from my laptop and leaving it on throughout the night. This has cleaned my log group in 30 hours or so.
Maybe log-minder can run continuously when it notices there is more work to do. That would be easier than running the CLI manually.
Bumps certifi from 2022.9.24 to 2022.12.7.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
It would be nice if we can filter out some Log Groups that do not have any retention and have a tag like - NoExpire= true as part of set-log-retention command.
We have some scenarios were we don't want to set retention for certain log groups, so filtering them out and setting default retention for other would be great.
cwlog-minder --dry-run set-log-retention --days 30 --filter
Great job with the tool!
It would be great if the tool tags the log groups automatically with
I used this as a workaround:
aws logs describe-log-groups | jq -r '.logGroups.logGroupName' | xargs -t -I% -n1 aws logs tag-log-group --tags Name=% --log-group-name=%
cloudwatch-logs utility cli aws-lambda aws