AWS Cloudwatch Log minder

AWS CloudWatch logs is an useful logging system, but it has two quircks. It does not allow you too set a default retention period for newly created log groups, and it does not delete empty log streams that are older than the retention period. This utility:

1. sets a default retention period on log groups without a period set.
2. removes empty log streams older than the retention period of the log group

You can use it as a command line utility. You can also install it as an AWS Lambda function and have your logs kept in order, NoOps style!

install the log minder

to install the log minder, type:

sh pip install aws-cloudwatch-log-minder

set default retention period

to set the default retention period on log groups without one, type: sh cwlog-minder --dry-run set-log-retention --days 30 This will show you which log groups will have its retention period set. Remove the --dry-run and it the retention period will be set. If you wish to set the retention of all log groups to the same value, type: sh cwlog-minder --dry-run set-log-retention --days 30 --overwrite

delete empty log streams

To delete empty log streams older than the retention period, type: sh cwlog-minder --dry-run delete-empty-log-streams This will show you which empty log streams will be deleted. Remove the --dry-run and these stream will be deleted.

deploy the log minder

To deploy the log minder as an AWS Lambda, type:

sh git clone https://github.com/binxio/aws-cloudwatch-log-minder.git cd aws-cloudwatch-log-minder aws cloudformation deploy \ --capabilities CAPABILITY_IAM \ --stack-name aws-cloudwatch-log-minder \ --template-file ./cloudformation/aws-cloudwatch-log-minder.yaml \ --parameter-overrides LogRetentionInDays=30 This will install the log minder in your AWS account and run every hour.

delete empty log groups

To delete empty log groups, type: sh cwlog-minder --dry-run delete-empty-log-groups This will show you which empty log groups will be deleted. Remove the --dry-run and these groups will be deleted. Do not use this command, if your log groups are managed by CloudFormation or Terraform.

verbose

sh export LOG_LEVEL=DEBUG cwlog-minder ...

region and profile selection

AWS regions and credential profiles can be selected via command line arguments or environment variables.

region via parameter

sh cwlog-minder --region eu-west-1 ...

region via environment

sh export AWS_DEFAULT_REGION=eu-west-1 cwlog-minder ...

profile via parameter

sh cwlog-minder --profile dev ...

profile via environment

sh export AWS_PROFILE=dev cwlog-minder ...

Issues

There are no log messages indicating success when a lot of log streams are deleted

opened on 2023-03-30 07:23:07 by 0xR

I was running log minder on a big collection of log steams. Since there was no log output I thought the tool was stuck or not working. When I turned on Debug logging I discovered it was doing a lot of log deletion. No log messages were shown because they were set to debug level.

Maybe a summary log message every 10 seconds can help show progress so that users don't think the script is not working.

Initial log stream deletion takes too long for older applications

opened on 2023-03-30 07:17:02 by 0xR

I have been running an application for more than 3 years without the log minder. There are >100k log streams in 1 log group. With the rate limits of AWS Cloudwatch and the default maximum log minder duration of 5 minutes it would take weeks to clean up all the log streams.

For now I've been running log-minder from my laptop and leaving it on throughout the night. This has cleaned my log group in 30 hours or so.

Maybe log-minder can run continuously when it notices there is more work to do. That would be easier than running the CLI manually.

Bump certifi from 2022.9.24 to 2022.12.7

opened on 2022-12-09 09:32:29 by dependabot[bot]

Bumps certifi from 2022.9.24 to 2022.12.7.

Commits

• 9e9e840 2022.12.07
• See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/binxio/aws-cloudwatch-log-minder/network/alerts).

Feature Request: Filter Log Groups by tag before setting retention

opened on 2021-04-13 16:45:17 by gogineni99

It would be nice if we can filter out some Log Groups that do not have any retention and have a tag like - NoExpire= true as part of set-log-retention command.

We have some scenarios were we don't want to set retention for certain log groups, so filtering them out and setting default retention for other would be great.

cwlog-minder --dry-run set-log-retention --days 30 --filter

Feature request: Tag LogGroup automatically with Name

opened on 2020-10-07 05:45:25 by estahn

Great job with the tool!

It would be great if the tool tags the log groups automatically with Name tag.

see https://github.com/guardian/cloudwatch-logs-management/issues/19

I used this as a workaround: aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | xargs -t -I% -n1 aws logs tag-log-group --tags Name=% --log-group-name=%