Tag: Amazon EC2

Scheduled shutdown of Amazon EC2 instances with the ability to postpone – User Guide

Nivlesh ChandraLeave a comment

Introduction

In my previous blog, I outlined the steps for deploying a solution that I had created, to shut down Amazon EC2 instances on a schedule. What makes my solution different from others is that, 15 minutes before the Amazon EC2 instance are scheduled to be turned off, an email will be sent to the owner, informing them of the upcoming scheduled shutdown, and giving them an opportunity to postpone the shutdown by 1 hour. The ability to easily postpone scheduled shutdowns temporarily has been extremely helpful to me, in situations where on those “once in a blue moon” nights, I found myself working through the night and didn’t want the server to suddenly be shut down due to a schedule.

In this blog, I will take the opportunity to provide instructions on how to use the shutdown scheduler solution.

Let’s get started.

Prerequisites

Actually, before we begin, let’s go through the things that this blog assumes to be already in place. These are required to help you in understanding the solution.

Here is a list of assumed items.

  1. The solution that was discussed in the previous blog – to shutdown Amazon EC2 instances on a schedule with the ability to postpone the shutdowns, has already been deployed in your own AWS Account. If it doesn’t exist, please follow the instructions in the previous blog to deploy it before continuing. The previous blog can be found at https://nivleshc.wordpress.com/2022/08/01/scheduled-shutdown-of-amazon-ec2-instances-with-the-ability-to-postpone-backend/
  2. The email address used as the FROM email address for Amazon Simple Email Service (SES) emails has already been verified. This is the EMAIL_FROM_ADDRESS variable in the Makefile, which is part of the deployment code. This is necessary, since without it, Amazon Simple Email Service (SES) will reject any emails that will be sent by this solution. From testing, I found that gmail email addresses work well as FROM email addresses for Amazon SES.
  3. You already have at least one Amazon EC2 instance provisioned in your AWS Account, which will be used to test the scheduled shutdown solution.

Instructions

Let’s begin.

  1. Login to your AWS Account via the AWS Management Console and then open the Amazon EC2 Portal.
  2. In the Amazon EC2 Portal, select the Amazon EC2 instance that you want to enable for auto shutdown. The lower part of the screen will display the configuration for that particular Amazon EC2 instance. Click on the Tags tab and then click on Manage tags. Add the following tags to this Amazon EC2 instance. Add them exactly as is described below, matching the case.
    1. owner_email – this is the email address to which all notifications (including those to postpone the scheduled shutdown) will be sent to regarding this specific Amazon EC2 instance.
    2. shutdown_time – this is a four digit time in HHMM format, in UTC timezone, for when this specific Amazon EC2 instance must be shutdown, it if is found running. This scheduled shutdown time applies to every day of the week. For example, to schedule this specific Amazon EC2 instance to be shutdown at 9:05 am UTC (7:05 pm AEST), set the shutdown time to 0905. You can use https://www.timeanddate.com/ to convert your local timezone to UTC.
    3. autoshutdown – this is a boolean that can be set to either True or False, to enable or disable the scheduled auto shutdown of this particular Amazon EC2 instance. For example, if you anticipate an Amazon EC2 instance to be running 24×7 for a couple of weeks, you could temporarily exclude it from the schedular by setting autoshutdown to False. This preserves your auto shutdown settings, however this Amazon EC2 instance will be ignored by the AutoShutdownEC2 AWS Lambda function. Once you are ready to re-enable the auto shutdown schedule, just set autoshutdown to True.
  3. Below is a screenshot of what the additional tags should look like.

4. Turn on the Amazon EC2 instance to which you just added the tags, if it is not already turned on.

5. Lets digress a bit.

When you start using Amazon Simple Email Service (SES) in a particular AWS Region, your Amazon SES account is put into a sandbox. This enables tighter controls, to ensure you don’t inadvertently send spam emails to others. For any email that you use with Amazon SES, be it the FROM address or the recipient email address, it all needs to be verified within Amazon SES. To do this, you will add these email addresses in Verified identities (this is under Configuration in the left-hand side menu in the Amazon SES dashboard). Amazon SES will then send an email with a verification link to that respective email address, asking the recipient to authorise the use of that email address with Amazon SES. If verification is not completed, Amazon SES will reject any emails sent to that email address (or for that email address to be used as a FROM address).

To find out if your Amazon SES account is in sandbox, open the Amazon SES dashboard and check for a message similar to that pasted below.

Having your Amazon SES account in a sandbox is not a big hinderance when testing the scheduled shutdown solution with a handful of Amazon EC2 instances. However this can quickly become an issue if the number of unique owner_email email addresses increase to a point that it is an administrative burden to verify each one of them. When this happens, you can create a support case to move your Amazon SES account out of sandbox. You will have to provide a valid reason for wanting to do this, however once AWS is satisfied, your account will be moved out of the sandbox. You can find the instructions for requesting your Amazon SES account to be taken out of sandbox here https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html.

6. The AutoShutdownEC2 AWS Lambda function runs every five minutes. Its output can be traced in Amazon CloudWatch Logs. Once the Amazon EC2 instance has been processed by the AutoShutdownEC2 AWS Lambda function, you will notice that a new tag has been added to it.

This new tag, actual_shutdown_time, is what AutoShutdownEC2 AWS Lambda function uses to track the time a particular Amazon EC2 instance needs to be shut down. This is the tag that will have its value updated, if the Amazon EC2 owner decides to postpone the scheduled shutdown for that particular day. Having a separate “housekeeping” tag ensures that we don’t have to change the shutdown_time tag, this keeps each day’s scheduled shutdown time the same. This means that the postponed shut down time will only apply to that particular day.

If the actual_shutdown_time is in the next 15 minutes, the AutoShutdownEC2 AWS Lambda function will send an email to the owner_email email address, informing them of the upcoming scheduled shutdown. The email will state the number of minutes left before the Amazon EC2 instance will be automatically shut down. In addition to that, the email will also contain a link to postpone the scheduled shutdown by 1 hour.

Below is an example of the email that will be sent out.

The AutoShutdownEC2 AWS Lambda also adds another tag email_notification_sent to the respective Amazon EC2 instance with a value of True. This is used to track if an email notification has already been sent to the owner. This ensures that the owner doesn’t get spammed by multiple notification emails regarding the upcoming scheduled shutdown.

If you need to postpone the shutdown, click on the Postpone shutdown by 1 hour button in the email. This will open the link in your default web browser, and connect to the postpone Amazon API Gateway endpoint.

Assuming that you clicked the button before the Amazon EC2 instance was automatically shutdown, you will see a message similar to below.

The actual_shutdown_time tag is also updated on the respective Amazon EC2 instance, to reflect the new shutdown time.

However, if you clicked the Postpone shutdown by 1 hour button after the Amazon EC2 instance was automatically shutdown, you will receive the following error message.

Note: You might be thinking, I could potentially use the same email to postpone the shutdown time the next time that Amazon EC2 instance is turned on. Well not really. A unique token is generated before the email notification is sent out with the postpone link. The unique token is added to an Amazon DynamoDB table, along with the details of that particular Amazon EC2 instance. This record is configured to expire in 15 minutes. The unique token is also added to the postpone link, and becomes part of the payload when you click the postpone button.

When you click on the link to postpone the shutdown time, the token is passed to the Amazon API Gateway endpoint, which in turn passes it to the PostponeEC2Shutdown AWS Lambda function. This AWS Lambda checks the Amazon DynamoDB table to confirm that the token it has received exists. If not then the above error message is shown.

If the token is valid, after the Amazon EC2 tags have been updated, the record for that token is deleted from the Amazon DynamoDB table. This ensures that the same token can’t be used more than once.

If the shutdown is not postponed, the Amazon EC2 instance will be automatically shutdown at the actual_shutdown_time. An email will also be sent to the owner, after the Amazon EC2 instance’s shutdown has started.

Below is an example of the email that will be sent out.

The AutoShutdownEC2 AWS Lambda also removes all the housekeeping tags. Below is a screenshot of the tags that remain on the Amazon EC2 instance after the cleanup. If you compare these to the tags that we started with (after we had added the additional tags), you will notice they are identical. This guarantees that the scheduled shutdown time remains the same every day, irrespective of the postponements that happen on a certain day.

If you find that you are having to postpone the shutdowns daily, this could mean that the configured scheduled shutdown time is no longer appropriate, in which case it might be worthwhile to update shutdown_time tag.

That’s pretty much it, in terms of how to use the shutdown schedular.

Some things to keep in mind

Below are some things to keep in mind when using the Amazon EC2 instance shutdown schedular.

  1. As mentioned before, the following tags are mandatory. Please ensure they exist for all Amazon EC2 instances that you want to enable for scheduled shutdowns.
    1. owner_email
    2. shutdown_time
    3. autoshutdown
  2. The solution uses the following housekeeping tags to keep track of things. Please do not modify or delete them.
    1. actual_shutdown_time
    2. email_notification_sent
  3. Below is the logic used to find out if a running Amazon EC2 instance enabled for a scheduled shutdown has gone beyond its shutdown time and needs to be turned off.
    1. the actual_shutdown_time tag contains the time that the Amazon EC2 instance needs to be turned off.
    2. If the AutoShutdownEC2 AWS Lambda function finds a running Amazon EC2 instance whose actual_shutdown_time is in the past and the difference between the current time and value of the actual_shutdown_time tag is less than the EC2ShutdownWindow minutes, that particular Amazon EC2 instance will be automatically shutdown. The reason for having a shutdown window is because the AutoShutdownEC2 AWS Lambda runs at discrete times, on a 5 minute interval. We need to be able to detect if an instance has passed its shutdown time, in between these invocations, and so should be turned off. However, we should not keep the window too large, so that if someone manually starts the instance after it had been automatically shut down, it won’t be automatically shut down again. The default value for EC2ShutdownWindow is 10 minutes. This can be modified in the Parameters section of template.yaml.
    3. Keep in mind that the AutoShutdownEC2 AWS Lambda function runs every 5 minutes. If you make EC2ShutdownWindow less than 5 minutes, for scenarios where the value in actual_shutdown_time tag has elapsed, based on the AutoShutdownEC2 AWS Lambda function’s invocation time, the difference between the current time and the value in actual_shutdown_time tag might or might not be less than the value of EC2ShutdownWindow, in which case, the Amazon EC2 instance will not be automatically shut down, as it should have been. Let me illustrate this with an example. Let’s say that you have set the EC2ShutdownWindow value to 3 minutes. The actual_shutdown_time value is 0905. The AutoShutdownEC2 AWS Lambda last ran at 0904. It will next run at 0909. At this next invocation, the difference between the current time and the value in actual_shutdown_time is 4. Unfortunately, this is more than EC2ShutdownWindow time, so the instance will not be turned off. This will inadvertently cost you unnecessarily. This is why it is highly recommended to set EC2ShutdownWindow to a value higher than the interval between two AutoShutdownEC2 AWS Lambda invocations.

I hope this blog has been useful in regards to giving you the information you need to use the Amazon EC2 instance shutdown schedular solution I created.

If you have any questions, feel free to leave your comments below and I will answer them as soon as I can.

Till the next time, stay safe!