Auto-remediate EBS volumes based on Disk Space with no downtime

Your production server can run out of space for a lot of reasons. This happens because an EC2 instance neither reports disk usage nor memory usage to AWS CloudWatch. It can be passed to CloudWatch through a script.

The two ways to auto-remediate 'DiskSpaceUtilization' is to either increase the EBS volume size or move the data.

You can simply adopt the following templates for this use case

Add DiskSpaceUtilization metric script on EC2 instances.

As disk space isn’t monitored, you need to add an additional metric on CloudWatch by sending an SSM command with a RunShellScript.

Increase EBS volume and expand partition.

Once your DiskSpaceUtilization is being tracked, you can set up CloudWatch alarms and SNS configurations.

Set up alarms for DiskSpaceUtilization and set SNS topic.

The CloudWatch alarm acts as a trigger to increase the EBS volume size by a default 20GB.

Delete a particular folder on EBS volume.

If you don’t want to expand the volume size (because it is costlier!), you can delete folders on the volume that are of no use to you, like junk scripts, and unused server software.