Migrate An Amazon Lex Bot From One AWS Account To Another AWS Account

Background

I recently had the need to migrate an Amazon Lex bot from one AWS Account to another. Assuming this to be a trivial task, requiring a configuration export from the source account and an import into the destination account, I was up for the task. Unfortunately, reality turned out to be quite different!

The reason? The Amazon Lex bot I intended to migrate was using an AWS Lambda function in its fulfillment. As I found out, Amazon Lambda functions are not included in the exported Amazon Lex bot configuration files. To make matters worse, even if you create the AWS Lambda function in the destination AWS Account, the import still failed as it was trying to access the AWS Lambda function in the source AWS Account!

In this blog I will provide instructions on how to successfully migrate an Amazon Lex bot that is using an AWS Lambda function for fulfillment, from one AWS Account to another. The same steps can be used to migrate Amazon Lex bots from one AWS Region to another AWS Region, within the same AWS Account.

Some additional information

When you create Amazon Lex bots, you can have one of the following fulfillment configurations:

  • (Scenario 1) Return parameters to client
  • (Scenario 2) AWS Lambda function

For the first configuration, migration is simple. Unfortunately for the second configuration, the migration is a bit more involved.

In this blog, I will cover migration patterns for both above-mentioned Amazon Lex bot configurations.

Scenario 1 – Migration Approach

The migration approach for this scenario is straight forward. Here are the steps:

  1. Login to the AWS Management Console and then go to the Amazon Lex portal.
  2. Change to the AWS Region where the Amazon Lex bot (the one that you want to migrate) is located.
  3. Then from the left-hand side menu, click Bots.
  4. From the right-hand side screen, click the bot that you want to export. This will open its configuration.
  5. From the top, click on the Settings tab. Then from the left-hand side menu, click General.
  6. In the right-hand side screen, browse to the IAM role and ensure it is AWSServiceRoleForLexBots. This is the default role for Amazon Lex bots. We need to confirm that no additional policies have been attached to this. Click the IAM role to open it.
  7. The Identity and Access Management portal will now open, showing the policies attached to the AWSServiceRoleForLexBots role. Confirm that only the AmazonLexBotPolicy IAM policy is attached. If there are any additional policies attached, note them down.
  8. Return to the Amazon Lex portal and select the bot that you want to migrate. From Actions click Export (Ensure your Amazon Lex bot has been published, otherwise you won’t be able to export it).
  9. In the next screen, choose the version of the bot you want to export and the platform you will be exporting to. Since the destination will be an Amazon Lex bot, choose Amazon Lex. Click Export.
  10. When prompted, provide a location to store the zip file containing the exported Amazon Lex bot configuration. Viola! The settings have now been successfully exported.
  11. To import, login to the AWS Management Console of the destination AWS Account and then go to the Amazon Lex portal.
  12. Change to the AWS Region where you want to import the Amazon Lex bot to.
  13. From the left-hand side menu, click Bots.
  14. Then from the right-hand side screen, click on Actions and choose Import.
  15. In the next screen, for Upload file browse to the configuration file that you had exported from the source AWS Account (listed above). Click Import.
  16. Once the import has finished, the bot will be displayed in the Amazon Lex portal.
  17. If the source Amazon Lex bot’s IAM role had additional policies attached, open the IAM role for the destination Amazon Lex bot and attach these additional IAM policies.
  18. Proceed to Build and then Publish the imported Amazon Lex bot.

You should now have a successfully migrated Amazon Lex bot in the destination account. Go through the configuration to confirm the settings and test it to ensure it behaves as expected. If you don’t need the source Amazon Lex bot, this can now be safely deleted.

Scenario 2 – Migration Approach

In my opinion, Scenario 1 is used mostly when you are learning about Amazon Lex bots. As you get more experienced with it and are ready to create bots for real-world applications, you will require an AWS Lambda function for fulfillment. Unfortunately, with such sophisticated Amazon Lex bots, the migration process is a bit more involved.

Here are the steps:

  1. Login to the AWS Management Console and then go to the Amazon Lex portal.
  2. Change to the AWS Region where the Amazon Lex bot (the one that you want to migrate) is located.
  3. Then from the left-hand side menu, click Bots.
  4. From the right-hand side screen, click the bot that you want to export. This will open its configuration.
  5. In the screen that is shown, ensure the Editor tab is selected.
  6. In the left-hand side screen, under Intents click on each intent that is listed, and note the Fulfillment configuration. If this is set to AWS Lambda function then export that AWS Lambda function’s configuration (Exporting AWS Lambda function configuration is out-of-scope for this blog, however as a guide, in the Editor tab of the Amazon Lex bot configuration, click on the View in Lambda console link underneath the Lambda function drop-down list. This will open the AWS Lambda function. Then from Actions click on Export function. You will be presented with two options – Download SAM file or Download deployment package. Based on your choice, you can then import the AWS Lambda function in the destination AWS Account using either AWS CloudFormation or AWS SAM).
  7. Go back to the Amazon Lex bot’s configuration. From the top, click on the Settings tab. Then from the left-hand side menu, click General.
  8. In the right-hand side screen, browse to the IAM role and ensure it is AWSServiceRoleForLexBots. This is the default role for Amazon Lex bots. We need to confirm that no additional policies have been attached to this. Click the IAM role to open it.
  9. The Identity and Access Management portal will now open, showing the policies attached to the AWSServiceRoleForLexBots role. Confirm that only the AmazonLexBotPolicy IAM policy is attached. If there are any additional policies attached, note them down.
  10. Go back to the Amazon Lex portal and in the right-hand side screen select the bot that is to be migrated.
  11. From Actions click Export (Ensure your Amazon Lex bot has been published, otherwise you won’t be able to export it).
  12. In the screen that comes up, select the version of the bot to export and the platform to export it to. Since the destination will be an Amazon Lex bot, for platform choose Amazon Lex. Click Export.
  13. When prompted, provide a location to store the zip file containing the exported Amazon Lex bot configuration. The settings have now been successfully exported.
  14. Login to the destination AWS Account and create the AWS Lambda functions that were exported in step 6 above. Ensure these are created in the same AWS Region as to where the migrated Amazon Lex bot will reside.
  15. Using the AWS Lambda portal, note down the ARN of all the AWS Lambda functions that were just created.
  16. As you might have guessed, the challenge with using the exported Amazon Lex bot configuration is that it still refers to the source account’s AWS Lambda functions. Due to this (unless you have cross-account permissions setup for AWS Lambda functions – however this is not what we want) the Amazon Lex bot import will fail. So, before we import the exported configuration, we need to update it so that it points to the newly created AWS Lambda functions.
  17. The exported configuration file is a zip file. Using an archiving utility such as winrar or winzip, unzip the exported configuration file. This will generate a JSON file containing the configuration.
  18. Open the JSON file in a code editor such as Microsoft Visual Studio Code.
  19. Locate the intents block. Inside this, find the fulfillmentActivity section. Inside this section, locate the uri key (underneath codeHook). The value for this uri key is the ARN of the AWS Lambda function that is being used by this intent for fulfilment. Update this uri value with the ARN of the AWS Lambda function that was just created in the destination AWS Account. Locate any other intents and update their uri values with the appropriate AWS Lambda function ARNs. Once done, save the JSON file.
  20. Pro Tip: If you want the name of the Amazon Lex bot in the destination AWS Account be different from the source Amazon Lex bot, change the value of the name key in the resource block of the JSON file.
  21. Using an archive utility (such as winrar or winzip) archive the updated JSON file into a zip file.
  22. Next, we need to grant invoke permissions to the Amazon Lex bot’s intent for the newly created AWS Lambda functions. If this is not in place, the import process will fail. There are two ways of achieving this. One is via the AWS Management Console, the other via AWS CLI. I will list both options in the next steps.
  23. Using AWS Management Console – In the destination AWS Account, go the Amazon Lex portal (ensure it is in the correct AWS Region) and create an Amazon Lex bot with the same name as the source Amazon Lex bot (this is just a dummy Amazon Lex bot). Then create an intent inside this Amazon Lex bot, give it the same name as the intent in the source Amazon Lex bot. Then change the fulfillment for this intent to AWS Lambda function and choose the appropriate newly created AWS Lambda function from the drop-down list. A message will be displayed stating that this will grant the intent invoke permissions on the AWS Lambda function. Click to confirm. It is extremely important that the dummy Amazon Lex bot name and the dummy intent name match the source Amazon Lex bot name and the intent.
  24. Using AWS CLI – use the following command in the destination AWS Account
    aws lambda add-permission --function-name <lambda-function-name> --action lambda:InvokeFunction --statement-id <statement-id-label> --principal lex.amazonaws.com --source-arn <intent-arn> --region <region>
    

    where

    lambda-function-name is the name of the AWS Lambda function for which the Amazon Lex bot intent will be granted invoke permissions.

    statement-id-label is just a label for this permission policy (shows as the value for sid).  As a good practice you can use the format lex-<region>-<intent-name> (for example lex-us-east-1-StartConversation).

    intent-arn is the ARN of destination Amazon Lex bot’s intent. However, as this doesn’t currently exist , you won’t be able to lookup this value. However, the ARN has the following format. Replace the placeholder variables with the appropriate values arn:aws:lex:<region>:<account-id>:intent:<intent-name>:* . Don’t forget to wrap the ARN value with double quotes when using in the CLI (for example “arn:aws:lex:us-east-1:11111022222:intent:StartConversation:*”).

    region is the region where the AWS Lambda function is.

  25. All the pre-requisites have now been completed. The import process can now be initiated. Go to the Amazon Lex portal and from the left-hand side menu click Bots.
  26. In the right-hand side screen, from Actions choose Import.
  27. In the next screen, for Upload file browse to the configuration file that you had exported from the source AWS Account (listed above). Click Import.
  28. If you had created a dummy Amazon Lex bot and intent (step 23), a prompt will appear asking to confirm that the intent can be overwritten. Click Overwrite and continue.
  29. Once the import process completes, you will have successfully created a replica of the Amazon Lex bot in the destination AWS Account.
  30. If the source Amazon Lex bot’s IAM role had additional policies attached, open the IAM role for the destination Amazon Lex bot and attach these additional IAM policies.
  31. Proceed to Build and then Publish the imported Amazon Lex bot.

You should now have a successfully migrated Amazon Lex bot in the destination account. Go through the configuration to confirm the settings and test it to ensure it behaves as expected. If you don’t need the source Amazon Lex bot, this can now be safely deleted.

I hope you found this blog useful. Till the next time, Enjoy!

Create an Omnichannel Chatbot using Amazon Lex and Amazon Connect

Background

These days, chatbots are used pretty much everywhere. From getting quotes to resetting password, their use cases are endless. They also elevate the customer experience, as now customers don’t need to call during your helpdesk’s manned hours, instead they can call anytime that is convenient to them. One of the biggest business benefits is that of lessening the load on their support staff.

A good practice to adhere by, when deploying chatbots is to make them channel agnostic. This means that your chatbots are available via the internet and also via a phone call, and they provide the same customer experience. This enables your customers to choose whichever channel suits them best, without any loss of customer experience.

In this blog, I will demonstrate how you can use Amazon Connect and Amazon Lex, to create an omnichannel chatbot.

I will be extending one of my previous blogs, so if you haven’t read it already, I would highly recommend that you do so, prior to continuing. Here is the link to the blog https://nivleshc.wordpress.com/2020/04/08/create-a-web-chatbot-for-generating-life-insurance-quotes-using-amazon-lex/).

High Level Architecture Diagram

Below is the high-level architecture diagram for the solution. We will build the section inside the blue box.

For steps 1 – 5 please refer to https://nivleshc.wordpress.com/2020/04/08/create-a-web-chatbot-for-generating-life-insurance-quotes-using-amazon-lex/

For steps 6 – 8 please refer to https://nivleshc.wordpress.com/2020/05/24/publish-a-web-chatbot-frontend-using-aws-amplify-console/

Steps 9 – 12 will be created in this blog and are described below:

9. The customer calls the phone number for the chatbot. This is attached to a contact flow in Amazon Connect

10. Amazon Connect proxies the customer to the Amazon Lex chatbot (this is the web chatbot created in https://nivleshc.wordpress.com/2020/04/08/create-a-web-chatbot-for-generating-life-insurance-quotes-using-amazon-lex/ )

11. The output from the Amazon Lex chatbot is sent back to Amazon Connect.

12. Amazon Connect converts the output from the Amazon Lex chatbot into audio and then plays it to the customer.

Prerequisites

This blog assumes the following:

  • you already have an Amazon Connect instance deployed and configured.
  • you have a working Amazon Lex web chatbot

You can refer to the following blogs, if you need to deploy either of the above prerequisites

https://nivleshc.wordpress.com/2020/04/08/create-a-web-chatbot-for-generating-life-insurance-quotes-using-amazon-lex/

https://nivleshc.wordpress.com/2020/05/24/publish-a-web-chatbot-frontend-using-aws-amplify-console/

Implementation

Let’s get started.

  1. Login to your AWS Management Console, open the Amazon Connect console and change to the respective AWS region.
  2. Within the Amazon Connect console, choose the instance that will be used and click its Instance Alias.
  3. In the next screen, from the left-hand side menu, click Contact flows. Then, in the right-hand side screen, under Amazon Lex select the Region where the Amazon Lex bot resides. From the Bot drop-down list, select the name of the Amazon Lex bot. Once done, click +Add Lex Bot.
  4. Click Overview from the left-hand side menu, and then click Login URL to open the Amazon Connect administration portal. Enter your administrator credentials (currently only the following internet browsers are supported – latest three versions of Google Chrome, Mozilla Firefox ESR, Mozilla Firefox).
  5. Once logged in, from the left hand-side menu, click Routing and then Contact flows.
  6. Click Create contact flow (located on the top-right). You will now see the contact flow designer.
  7. Enter a name for the contact flow (top left where it says Enter a name)
  8. From the left-hand side menu, expand Interact and drag Get customer input to the right-hand side screen.
  9. Click on the circle to the right of Start in the Entry point block and drag the arrow to the Get customer input block. This will connect the two blocks.
  10. Click on the title of the Get customer input block to open its configuration.
  11. Select Text-to-speech or chat text. Click Enter text and in the textbox underneath, type the message you want to play to the customer when they call the chatbot.
  12. Click Amazon Lex and then under Lex bot Name select the Amazon Lex bot that you had created earlier (if the bot doesn’t show, ensure you had carried out step 3 above). Under Intents type the Amazon Lex bot intent that should be invoked. Click Save.

13. From the left-hand side menu, expand Interact and drag two Play prompt blocks to the right-hand side screen.

14. The first Play prompt block will be used to play a goodbye message, after the Amazon Lex bot has finished execution. Click on the title of this block to open its configuration. Click Text-to-speech or chat text and then click Enter text. Enter a message to be played before the call is ended. Click Save.

  15. The second Play prompt block will be used to play a message when an error occurs. Click on the title of this block to open its configuration. Click Text-to-speech or chat text and then click Enter text. Enter a message to be played when an error occurs. Click Save.

16. From the left-hand side menu, expand Terminate / Transfer and drag the Disconnect / hang up block to the right-hand side screen.

17. In the designer (right-hand side screen), in the Get customer input block, click the circle beside startConversation (this is the name of your Amazon Lex bot intent) and drag the arrow to the first Play prompt block.

18. Repeat step 17 for the circle beside Default in the Get customer input block.

19. In the Get customer input block, click the circle beside Error and drag it to the second Play prompt block.

20. Inside both the Play prompt blocks, click the circle beside Okay and drag the arrow to the Disconnect / hang up block.

21. From the top-right, click Save. This will save the work you have done.

22. From the top-right, click Publish. You will get a prompt Are you sure you want to publish this content flow? Click Publish.

23. Once done, you should see a screen similar to the one below:

24. Next, we need to ensure that whenever someone calls, the newly created contact flow is invoked. To do this, from the left-hand side menu, click Routing and then click Phone numbers.

25. In the right-hand side, click the phone number that will be used for the chatbot. This will open its settings. Enter a description (optional) and then from the drop-down list underneath Contact flow / IVR, choose the contact flow that was created above. Click Save.

Give it a few minutes for the settings to take effect. Now, call the phone number that was assigned to the contact flow above. You should be greeted by the welcome message you had created above. The phone chatbot experience would be the same as what you experienced when interacting with the chatbot over the internet!

Congratulations! You just created your first omnichannel chatbot! How easy was that?

Till the next time, Enjoy!

Publish A Web Chatbot Frontend Using AWS Amplify Console

Background

In my previous blog (https://nivleshc.wordpress.com/2020/04/08/create-a-web-chatbot-for-generating-life-insurance-quotes-using-amazon-lex/), I demonstrated how easy it was to create a web chatbot using Amazon Lex. As discussed in that blog, one of the challenges with Amazon Lex is not having an out-of-the-box frontend solution for the bots. This can throw a spanner in the works, if you are planning on showcasing your chatbots to customers, without wanting to overwhelm them with the code. Luckily, with some work, you can create a front-end that exposes just the bot. I provided instructions for achieving this in the same blog.

Having a static website hosted out of an Amazon S3 bucket is good, however it does come with a few challenges. As the website gains popularity, it becomes more integral to your business. Soon, you will not be able to afford any website outages. In such situations, how do you deploy changes to the website without breaking it? How do you track the changes, to ensure they can be rolled back, if something does break? How do you ensure your end-users don’t suffer from slow webpage loads? These are some of the questions that need to be answered, as your website achieves popularity.

AWS Amplify Console provides an out-of-the-box solution for deploying static websites. The contents of the website can be hosted in a source code repository. This provides an easy solution to track changes, and to rollback, should the need arise. AWS Amplify Console serves the website using Amazon CloudFront, AWS’s Content Delivery Network. This ensures speedy page loads for end-users. These are just some of the features that make hosting a static website using AWS Amplify Console a great choice.

In this blog, I will modify my life Insurance quote web chatbot solution, by migrating its frontend from an Amazon S3 bucket to AWS Amplify Console.

High Level Architecture Diagram

Below is a high-level architecture diagram for the solution described in this blog.

Steps 1 – 5 are from my previous blog https://nivleshc.wordpress.com/2020/04/08/create-a-web-chatbot-for-generating-life-insurance-quotes-using-amazon-lex

Steps 6 – 8 will be created in this blog and are described below:

6. Website developer will push changes for the Amazon Lex web chatbot frontend to GitHub.

7. GitHub will inform AWS Amplify about the new changes.

8. AWS Amplify will retrieve the changes from GitHub, build the new web chatbot frontend and deploy it, thereby updating the previous web chatbot        frontend.

Implementation

Before we continue, if you haven’t already, I would highly recommend that you read my Amazon Lex Life Insurance quote generating web chatbot blog ( https://nivleshc.wordpress.com/2020/04/08/create-a-web-chatbot-for-generating-life-insurance-quotes-using-amazon-lex)

Let’s get started.

  1. Login to your AWS Management Console, open the AWS Amplify Console and change to your AWS Region of choice.
  2. Click on Connect app.
  3. Next, choose the location where the source code is hosted. As I have stored the web chatbot frontend files in GitHub, I chose GitHub. Note that the repository should contain only the frontend files. Click Continue.
  4. Unless you had previously authorised AWS Amplify Console access to your source code repository, a screen will pop-up requesting access to your source code repository. Click Authorize aws-amplify-console.
  5. Once successfully authorised, you will be returned to the AWS Amplify Console. Using the dropdown menu beside Select a repository select the repository that contains the frontend code.
  6. Next, choose the Branch to use and then click Next.
  7. The next screen shows the Configure build settings page. AWS Amplify Console will inspect the source code and deduce the appropriate App build and test settings. If what is shown is incorrect, or you would like to modify it, you can use the Edit button.

    In my case, I did not find anything needed changing from what AWS Amplify Console had provided.

    You can change the App name, if this needs to be different from what is automatically provided.

    Click Next.

  8. In the next screen, review all the settings. Once confirmed, click Save and deploy.
  9. AWS Amplify Console will start creating the application. You will be redirected to the application’s configuration page, where on the right, a continuous deployment pipeline, similar to the one below, will be shown.

  10. Wait for all stages of the pipeline to complete successfully and then click on the url on the left (the one similar to https://master…amplifyapp.com). The page that opens next is the Insurance Chatbot Frontend, served by AWS Amplify Console! (below is how the web chatbot looks like)

  11. Now, whenever the frontend files are modified and pushed into the master branch of the source code repository (GitHub in this case), AWS Amplify Console will automatically update the Insurance Chatbot frontend, with all changes easily trackable from within GitHub.
  12. You can use a custom domain name, to make the application URL more personalised (by default, AWS Amplify Console applications are allocated the amplifyapp.com domain urls). To do this, from your application’s configuration page, click Domain management in the left-hand side menu. Then click add domain and follow the instructions.
  13. You might also benefit from email notifications whenever AWS Amplify Console updates your application. To configure this, from your application’s configuration page, click Notifications in the left-hand side menu. Then click Add notification and add an email address to receive notifications for successful and failed builds.
  14. To view site access logs, from your applications configuration page, click Access logs in the left-hand side menu.

There you go. Hopefully this provides valuable information for those looking for an easy solution to deploy their static websites in a consistent, auditable and highly available manner.

Till the next time, Enjoy!

Automate Training, Build And Deployment Of Amazon SageMaker Models Using AWS Step Functions

Background

A few weeks back, I was tasked with automating the training, build and deployment of an Amazon SageMaker model. Initially, I thought that an AWS Lambda Function would be the best candidate for this, however as I started experimenting, I quickly realised that I needed to look elsewhere.

After some research, I found articles that pointed me towards AWS Step Functions. As it happens, AWS has been making AWS Step Functions more Amazon SageMaker friendly, to the point that AWS Step Functions now natively supported most of the Amazon SageMaker APIs.

With the technology decided, I started figuring out how I would achieve what I had set out to do. I did find some good documentation and examples, however they didn’t entirely cover everything I was after.

After much research and experimentation, I finally created a solution that was able to automatically train, build and then deploy an Amazon SageMaker model.

In this blog, I will outline the steps I followed, with the hope that it benefits those wanting to do the same, saving them countless hours of frustration and experimentation.

High Level Architecture Diagram

Below is a high-level architecture diagram of the solution I used.

The steps (as denoted by the numbers in the diagram) are as follows:

  1. The first AWS Step Function state calls the Amazon SageMaker API to create a Training Job, passing it all the necessary parameters.
  2. Using the supplied parameters, Amazon SageMaker downloads the training and validation files from the Amazon S3 bucket, and then runs the training algorithm. The output is uploaded to the same Amazon S3 bucket that contained that training and validation files.
  3. The next AWS Step Function state calls the Amazon SageMaker API to create a model,  using the artifacts from the Training Job.
  4. The next AWS Step Function state calls the Amazon SageMaker API to create an endpoint configuration, using the model that was created in the previous state.
  5. The next AWS Step Function state calls the Amazon SageMaker API to create a model endpoint, using the endpoint configuration that was created in the previous state.
  6. Using the endpoint configuration, Amazon SageMaker deploys the model using Amazon SageMaker Hosting Services, making it available to any client wanting to use it.

Let’s get started.

Implementation

For this blog, I will be using the data and training parameters described in the Amazon SageMaker tutorial at https://docs.aws.amazon.com/sagemaker/latest/dg/gs-console.html

1. Create an Amazon S3 bucket. Create a folder called data inside your Amazon S3 bucket, within which, create three subfolders called train, validation and test (technically these are not folders and subfolders, but keys. However, to keep things simple, I will refer to them as folders and subfolders).

2. Follow Step 4 from the above-mentioned Amazon SageMaker tutorial (https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-preprocess-data.html) to download and transform the training, validation and test data. Then upload the data to the respective subfolders in your Amazon S3 bucket (we won’t be using the test data in this blog, however you can use it to test the deployed model).

For the next three months, you can download the transformed training, validation and test data from my Amazon S3 bucket using the following URLs

https://niv-sagemaker.s3-ap-southeast-2.amazonaws.com/data/train/examples

https://niv-sagemaker.s3-ap-southeast-2.amazonaws.com/data/validation/examples

https://niv-sagemaker.s3-ap-southeast-2.amazonaws.com/data/test/examples

3. Create an AWS IAM role with the following permissions
AmazonSageMakerFullAccess (AWS Managed Policy)

and a custom policy to read, write and delete objects from the Amazon S3 bucket created in Step 1. The policy will look similar to the one below

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucketName"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucketName/*"
]
}
]
}

where bucketName is the name of the Amazon S3 bucket created in Step 1 above.

4. Open the AWS Step Functions console and change to the AWS Region where the Amazon SageMaker model endpoint will be deployed.

5. Create a new state machine, choose Author with code snippets and set the type to Standard.

6. Under Definition delete everything and paste the following
{
"Comment": "An AWS Step Function State Machine to train, build and deploy an Amazon SageMaker model endpoint",
"StartAt": "Create Training Job",

The above commands provide a comment describing the purpose of this AWS Step Function state machine and sets the first state name as Create Training Job.

For a full list of Amazon SageMaker APIs supported by AWS Step Functions, please refer to https://docs.aws.amazon.com/step-functions/latest/dg/connect-sagemaker.html

7. Use the following code to create the first state (these are the training parameters described in the above-mentioned Amazon SageMaker tutorial).

"States": {
"Create Training Job": {
"Type": "Task",
"Resource": "arn:aws:states:::sagemaker:createTrainingJob.sync",
"Parameters": {
"TrainingJobName.$": "$$.Execution.Name",
"ResourceConfig": {
"InstanceCount": 1,
"InstanceType": "ml.m4.xlarge",
"VolumeSizeInGB": 5
},
"HyperParameters": {
"max_depth": "5",
"eta": "0.2",
"gamma": "4",
"min_child_weight": "6",
"silent": "0",
"objective": "multi:softmax",
"num_class": "10",
"num_round": "10"
},
"AlgorithmSpecification": {
"TrainingImage": "544295431143.dkr.ecr.ap-southeast-2.amazonaws.com/xgboost:1",
"TrainingInputMode": "File"
},
"OutputDataConfig": {
"S3OutputPath": "s3://bucketName/data/modelartifacts"
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 86400
},
"RoleArn": "iam-role-arn",
"InputDataConfig": [
{
"ChannelName": "train",
"ContentType": "text/csv",
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://bucketName/data/train",
"S3DataDistributionType": "FullyReplicated"
}
}
},
{
"ChannelName": "validation",
"ContentType": "text/csv",
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://bucketName/data/validation",
"S3DataDistributionType": "FullyReplicated"
}
}
}
]
},
"Retry": [
{
"ErrorEquals": [
"SageMaker.AmazonSageMakerException"
],
"IntervalSeconds": 1,
"MaxAttempts": 1,
"BackoffRate": 1.1
},
{
"ErrorEquals": [
"SageMaker.ResourceLimitExceededException"
],
"IntervalSeconds": 60,
"MaxAttempts": 1,
"BackoffRate": 1
},
{
"ErrorEquals": [
"States.Timeout"
],
"IntervalSeconds": 1,
"MaxAttempts": 1,
"BackoffRate": 1
}
],
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.cause",
"Next": "Display Error"
}
],
"Next": "Create Model"
},

I would like to call out a few things from the above code

“Resource”: “arn:aws:states:::sagemaker:createTrainingJob.sync” refers to the Amazon SageMaker API for creating a Training Job. When this state task runs, you will be able to see this Training Job in the Amazon SageMaker console.

TrainingJobName is the name given to the Training Job and it must be unique within the AWS Region, in the AWS account. In my code, I am setting this to the Execution Name (internally referred to as $$.Execution.Name), which is an optional parameter that can be supplied when executing the AWS Step Function state machine. By default, this is set to a unique random string, however to make the Training Job name more recognisable, provide a more meaningful unique value when executing the state machine. I tend to use the current time in the format <training-algorithm>-<year><month><date>-<hour><minute><second>

If you have ever used Jupyter notebooks to run an Amazon SageMaker Training Job, you would have have used a line similar to the following:

        container = get_image_uri(boto3.Session().region_name, ‘xgboost’)

Yes, your guess is correct! Amazon SageMaker uses containers for running Training Jobs. The above assigns the xgboost training algorithm container from the region that the Jupyter notebook is running in.

These containers are hosted in Amazon Elastic Container Registry (Amazon ECR) and maintained by AWS. For each training algorithm that Amazon SageMaker supports, there is a specific container. Details for these containers can be found at https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html.

When submitting a Training Job using AWS Step Functions, you must supply the correct container name, from the correct region (the region where you will be running Amazon SageMaker from). This information is passed using the parameter TrainingImage. To find the correct container path, use https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html.

Another method for getting the value for TrainingImage is to manually submit a Training Job using the Amazon SageMaker console, using the same training algorithm that you will be using in the AWS Step Function state machine. Once the job has started, open it and have a look under the section Algorithm. You will find the Training Image for that particular training algorithm, for that region, listed there. You can use this value for TrainingImage.

S3OutputPath is the location where Amazon SageMaker will store the model artifacts after the Training Job has successfully finished.

RoleArn is the ARN of the AWS IAM Role that was created in Step 3 above

S3Uri under ChannelName: train is the Amazon S3 bucket path to the folder where the training data is located

S3Uri under ChannelName:validation is the Amazon S3 bucket path to the folder where the validation data is located

DON’T FORGET TO CHANGE bucketName TO THE AMAZON S3 BUCKET THAT WAS CREATED IN STEP 1 ABOVE

In the next AWS Step Function state, the model will be created using the artifacts generated from the Training Job.

8. An AWS Step Function state receives input parameters, does its processing and then produces an output. If there is another state next in the path, the output is provided as an input to that state. This is an elegant way for passing dynamic information between states.
Here is the code for the next AWS Step Functions state.
"Create Model": {
"Parameters": {
"PrimaryContainer": {
"Image": "544295431143.dkr.ecr.ap-southeast-2.amazonaws.com/xgboost:1",
"Environment": {},
"ModelDataUrl.$": "$.ModelArtifacts.S3ModelArtifacts"
},
"ExecutionRoleArn": "iam-role-arn",
"ModelName.$": "$.TrainingJobName"
},
"Resource": "arn:aws:states:::sagemaker:createModel",
"Type": "Task",
"ResultPath":"$.taskresult",
"Next": "Create Endpoint Config"
},

Image refers to the same container that was used in the Create Training Job state.

ModelDataUrl refers to the location where the model artifacts that were created in the previous state are stored. This value is part of the output (input to this state) from the previous state. To reference it, use $.ModelArtifacts.S3ModelArtifacts

ExecutionRoleArn is the ARN of the AWS IAM Role that was created in Step 3 above.

“Resource”: “arn:aws:states:::sagemaker:createModel” refers to the Amazon SageMaker API for creating a model

To keep things simple, the name of the generated model will be set to the TrainingJobName. This value is part of the output (input to this state) from the previous state. To reference it, use $.TrainingJobName

After this state finishes execution, you will be able to see the model in the Amazon SageMaker console.

The next state is for creating an Endpoint Configuration using the model that was just created.

Before we continue, I want to point out an additional parameter that I am using “ResultPath”:”$.taskresult”. Let me explain the reason for using this. In my next state, I must provide the name of the model that will be used to create the Endpoint Configuration. Unfortunately, this name is not part of the output of the current state Create Model, so I won’t be able to reference it. However, as you might remember, for simplicity, we set the model name to be the same as TrainingJobName and guess what, this is part of the current states input parameters! Now, if only there was a way for me to make the current state to include its input parameters in its output. Oh wait! There is a way. Using the command  “ResultPath”:”$.taskresult” instructs this AWS Step Function state to include its input parameters in its output.

9. Here is the code for the AWS Step Function state to create an Endpoint Config.

"Create Endpoint Config": {
"Type": "Task",
"Resource": "arn:aws:states:::sagemaker:createEndpointConfig",
"Parameters":{
"EndpointConfigName.$": "$.TrainingJobName",
"ProductionVariants": [
{
"InitialInstanceCount": 1,
"InstanceType": "ml.t2.medium",
"ModelName.$": "$.TrainingJobName",
"VariantName": "AllTraffic"
}
]
},
"ResultPath":"$.taskresult",
"Next":"Create Endpoint"
},

This state is pretty straight forward.

“Resource”: “arn:aws:states:::sagemaker:createEndpointConfig” refers to the Amazon SageMaker API to create an Endpoint Configuration

For simplicity, we will set the Endpoint Configuration name to be the same as the TrainingJobName. The Endpoint will be deployed initially using one  ml.t2.medium instance.

As in the previous state, we will use “ResultPath”:”$.taskresult” to circumvent the lack of parameters in the output of this state.

In the final state, I will instruct Amazon SageMaker to deploy the model endpoint.

10. Here is the code for the final AWS Step Function state.

"Create Endpoint":{
"Type":"Task",
"Resource":"arn:aws:states:::sagemaker:createEndpoint",
"Parameters":{
"EndpointConfigName.$": "$.TrainingJobName",
"EndpointName.$": "$.TrainingJobName"
},
"End": true
},

The Endpoint Configuration from the previous state is used to deploy the model endpoint using Amazon SageMaker Hosting Services.

“Resource”:”arn:aws:states:::sagemaker:createEndpoint” refers to the Amazon SageMaker API for deploying an endpoint using Amazon SageMaker Hosting Services. After this state completes successfully, the endpoint is visible in the Amazon SageMaker console.

The name of the Endpoint, for simplicity is set the same as TrainingJobName

To keep things tidy, it is nice to display an error when things don’t go as planned. There is an AWS Step Function state for that!

11. Here is the code for the state that displays the error message. This state only gets invoked if there is an error in the Create Training Job state.

"Display Error":{
"Type": "Pass",
"Result": "Finished with errors. Please check the individual steps for more information",
"End": true
}

The full AWS Step Function state machine code is available at  https://gist.github.com/nivleshc/a4a99a5c2bca1747b6da0d7da0e388c1

When creating the AWS Step Function state machine, you will be asked for an AWS IAM Role that will be used by the state machine to run the states. Unless you already have an AWS IAM Role that can carry out all the state tasks, choose the option to create a new AWS IAM Role.

To invoke the AWS Step Function state machine, just click on new execution and provide a name for the execution id. As each of the states are run, you will see the visual feedback in the AWS Step Function schematic. You will be able to see the tasks in the Amazon SageMaker console as well.

To take the above one step further, you could invoke the AWS Step Function state machine whenever new training and validation data is available in the Amazon S3 bucket. The new model can then be used to update the existing model endpoint.

Thats it folks! This is how you can automatically train, build and deploy an Amazon SageMaker model!

Once you are finished, don’t forget to clean-up, to avoid any unnecessary costs.

The following must be deleted using the Amazon SageMaker console

  • The model endpoint
  • The Endpoint Configuration
  • The model
  • Any Jupyter Notebook instances you might have provisioned and don’t need anymore
  • Any Jupyter notebooks that are not needed anymore

If you don’t have any use for the following, these can also be deleted.

  • The contents of the Amazon S3 bucket and the bucket itself
  • The AWS IAM Role that was created and the custom policy to access the Amazon S3 bucket and its contents
  • The AWS Step Function state machine

Till the next time, Enjoy!

Create A Web Chatbot For Generating Life Insurance Quotes Using Amazon Lex

Background

A few weeks back, I was asked to create a proof of concept web based chatbot for one of our clients. The web chatbot was to be used for generating life insurance quotes. The requirements were quite simple: ask a customer a few critical questions, use the responses to approximate their yearly premium and then display the premium on a webpage. Simple!

I don’t like reinventing the wheel and where possible, I leverage existing AWS services. For the task at hand, I decided to use Amazon Lex.

This blog provides the instructions for creating a web-based life insurance quote generating chatbot. It also highlights some of the challenges I faced while going from ideation to the finished product.

Let’s begin.

High Level Architecture

Below is a high-level overview of what I built.

  1. The customer will browse to the chatbot website.
  2. The customer will invoke the Amazon Lex Bot.
  3. The Amazon Lex Bot will ask a few questions and then pass the responses to an AWS Lambda function, to approximate the yearly premium for the customer.
  4. The response from the AWS Lambda function will be passed back to the Amazon Lex Bot.
  5. The Amazon Lex Bot will display the yearly premium estimate on the chatbot website.

Implementation

Let’s build the various components, as shown in the high-level overview.

AWS Lambda Function

When the Amazon Lex Bot invokes AWS Lambda, it actually calls the lambda_handler function and passes all the relevant parameters. The AWS Lambda will then use the supplied information to calculate the yearly premium and return the result.

I have pasted below the AWS Lambda Python 3.7 code that I used (getLifeInsuranceQuote). Do pay attention to the format of the return value from the AWS Lambda function. This is the format that Amazon Lex expects. To estimate the yearly premium, my AWS Lambda function called a machine learning model that had been pre-trained with life insurance data.

import json
from botocore.vendored import requests
from dateutil.relativedelta import relativedelta
from datetime import datetime
def lambda_handler(event, context):
print("event:"+str(event))
print("context:"+str(context))
customer_state = event['currentIntent']['slots']['State']
customer_firstname = event['currentIntent']['slots']['FirstName']
customer_lastname = event['currentIntent']['slots']['LastName']
customer_dob_str = event['currentIntent']['slots']['DOB']
customer_coverlevel = event['currentIntent']['slots']['CoverLevel']
customer_smoker = event['currentIntent']['slots']['Smoker']
customer_gender = event['currentIntent']['slots']['Gender']
print(customer_state)
print(customer_firstname)
print(customer_lastname)
print(customer_dob_str)
print(customer_coverlevel)
print(customer_smoker)
print(customer_gender)
date_now = datetime.now()
date_now_year = date_now.year
customer_dob_year = int(customer_dob_str)
customer_age = date_now_year customer_dob_year
print("Customer YOB:" + customer_dob_str)
print("Customer age:" + str(customer_age))
if customer_gender == "Female":
sex = 0
else:
sex = 1
if customer_smoker == "NO":
smoker = 0
else:
smoker = 1
url = "urlformlmodelapi
data = {"age": customer_age, “state”: customer_state, "sex": sex, "smoker": smoker}
r = requests.post(url,json=data)
premium = r.json()['claim_pred']
print("premium: " + str(premium))
message = customer_firstname + " from what you have told me, your monthly premiums will be approximately $" + str(round(premium/12))
return {
"sessionAttributes": {},
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "PlainText",
"content": message
}
}
}

Amazon Lex Bot

In this section, I will take you through the steps to create the Amazon Lex bot.

  1. Sign into the AWS console and then browse to the Amazon Lex service page
  2. On the left-hand side of the screen, click on Bots and then from the right-hand side, click Create.
  3. In the next screen, click Custom bot.
  4. Give the bot a name (I called mine LifeInsuranceBot) and set Output voice to None. This is only a text-based application.
  5. Set the Session timeout to 5 minutes.
  6. Leave Sentiment Analysis set to No. Leave the IAM role set to the default settings. Set COPPA to No.
  7. Click Create. The Amazon Lex bot will now be created for you.
  8. In the next screen, click on the Editor tab from the top menu.

    Before we continue, let’s go over some terminology that is used by Amazon Lex.

    Intents – an intent, in its simplest form, encapsulates what you are trying to achieve. In our case, the intent is to generate a life insurance quote.
    Utterances – this describes the possible phrases that a user could use to invoke an intent. Amazon Lex uses utterances to identify which intent it should pass the request to. In our situation, a possible utterance would be “I would like to get a life insurance quote”.
    Slots – these can be thought of as variables. You would ask the user a question and the response will be held in a slot. Like variables, a slot must have a type. The slot type is used by Amazon Lex to convert the user’s response into the correct format. For example, if you ask the user for their date of birth, the slot that will capture their response must have a type of AMAZON.DATE. This ensures that the date of birth is stored as a date.

  9. From the left-hand side menu, click on the plus sign beside Intents and then click Create Intent. You will be asked for a unique name for your intent. In my case, I set the intent name to generateLifeInstanceQuote.
  10. In the right-hand side, under Sample utterances enter phrases that will invoke this intent. I set this to I would like to get a life insurance quote.
  11. Amazon Lex comes with a lot of built-in slot types, however if they don’t match your use case, you can easily create custom slot types. For our intent, we will create three custom slot types.

    From the left-hand side, click on the plus beside Slot types and then create new slot types as per the following screenshots

  12. On to the Slots! As I mentioned previously, slots are used by Amazon Lex to get responses from the user. Each slot has a name, a type and a prompt. The prompt is what Amazon Lex will ask the user, and the response is stored in that particular slot. The prompts are asked in the order of priority assigned to them, from lowest to highest. I prefer to give some “character” to my prompts, to keep users engaged. ProTip: You can reference other slots in your prompt by enclosing the slot name within {} For example, if you are capturing the user’s name in the slot name FirstName, then you can prompt the user with Hello {FirstName}, how are you today?. When Amazon Lex prompts the user, it will insert the user’s name in place of FirstName. A touch of personalisation with minimal effort!
    Create slots as per the following screenshot.The slot type for FirstName is AMAZON.US_FIRST_NAME and the prompt is “Ok, I can help you with that. Let me get some details first. What is your first name?”

    The slot type for LastName is AMAZON.US_LAST_NAME

    The prompt for DOB is “Thanks {FirstName}. What year were you born in?”

    The prompt for CoverLevel is “Thank you for answering the questions {FirstName}. What amount do you want to take out the life insurance for?”

  13. The responses from the user will be passed to the AWS Lambda function, the result provided to the user.

    To enable this, in the right-hand side, under Fulfillment select AWS Lambda function and choose the AWS Lambda function that was created from the drop down beside Lambda function. Choose the appropriate version or alias.

  14. Click Save Intent.
  15. To get the Bot ready, click on Build from the top right-hand side of the screen.
  16. After the build is complete, you can test the Bot by using the Test Chatbot console from the top right-hand side of the screen.

Time to Panic!!!

Having successfully tested the Amazon Lex Bot, I was quite impressed with myself. But wait! I couldn’t find any way to “publish” it to a website! I didn’t want to showcase this Bot by using the “Test Chatbot” console! This is when I started panicking!

A Life Saver!

After searching frantically, I came across https://aws.amazon.com/blogs/machine-learning/greetings-visitor-engage-your-web-users-with-amazon-lex/, this article had exactly what I needed, a way to integrate my Amazon Lex bot with a html front end! Yay!

Amazon Cognito Identity Pool

    1. Go to the Amazon Cognito service page and then click on Manage Identity Pools. Then click on Create new identity pool.
    2. Provide a name for the Identity pool (I named mine LifeInsuranceBotPool) and tick the option Enable access to unauthenticated identities and click Create Pool.
    3. In the next screen, you will be asked to assign an IAM role. Click on View Details and note down the name of the roles that will be created. Then click Allow.
    4. The identity pool will be created and in the next screen, a sample code with the IdentityPoolId will be shown. Change the platform to Javascript and note down the IdentityPoolId and the region.
    5. Go to the AWS IAM service page and locate the two IAM roles that Amazon Cognito had created. Open each one of them and attach the following additional policies:
      • AmazonLexRunBotsOnly
      • AmazonPollyReadOnlyAccess

A front-end for our Amazon Lex Bot

The front end will be a static html page, served from an Amazon S3 bucket.

To make things easier, I have extracted the html code for the static website from  https://aws.amazon.com/blogs/machine-learning/greetings-visitor-engage-your-web-users-with-amazon-lex/ .

It is available at https://gist.github.com/nivleshc/bff75e30cc4f0133aab3abde8248814f

Save the above file as index.html and then carry out the following modifications.

Locate the following lines of code in index.html (lines 68 – 97)

// Initialize the Amazon Cognito credentials provider
AWS.config.region = 'us-east-1'; // Region
AWS.config.credentials = new AWS.CognitoIdentityCredentials({
// Provide your Pool Id here
IdentityPoolId: 'us-east-1:XXXXX',
});
var lexruntime = new AWS.LexRuntime();
var lexUserId = 'chatbot-demo' + Date.now();
var sessionAttributes = {};
function pushChat() {
// if there is text to be sent…
var wisdomText = document.getElementById('wisdom');
if (wisdomText && wisdomText.value && wisdomText.value.trim().length > 0) {
// disable input to show we're sending it
var wisdom = wisdomText.value.trim();
wisdomText.value = '…';
wisdomText.locked = true;
// send it to the Lex runtime
var params = {
botAlias: '$LATEST',
botName: 'BookTrip',
inputText: wisdom,
userId: lexUserId,
sessionAttributes: sessionAttributes
};

Change the values for AWS.config.region and AWS.config.credentials to what was displayed in the sample code when the Amazon Cognito Identity pool was created.

Replace chatbot-demo in the variable lexUserId to something more descriptive. This will be the user created within your Amazon Cognito Identity pool whenever the static website is accessed (for my deployment, I set this to lifeinsurancebot).

Change the botName to the name of your Amazon Lex Bot (in my case, my Amazon Lex Bot was called LifeInsuranceBot). If you have multiple versions of your Amazon Lex Bot and you are not using the latest version, then change the variable botAlias to the version you are using.

You might have noticed that index.html contains a lot of references to the demo chatbot that was created in https://aws.amazon.com/blogs/machine-learning/greetings-visitor-engage-your-web-users-with-amazon-lex/. I would suggest going through the html code and changing these references so that they refer to your own Amazon Lex Bot.

Next, we need a html page that will be returned when an error occurs with our static website. As good chefs do, I prepared one earlier. Download the contents of https://gist.github.com/nivleshc/853c7efc7979bdff6b5cc1a49074b9ce and save it as error.html.

Follow the steps below to create an Amazon S3 hosted static website

  1. Create an Amazon S3 bucket in a region closest to your Amazon Lex Bot (it is highly recommended to have the Amazon Lex Bot in an AWS region that is closest to your users)
  2. Upload the two files from above (index.html and error.html) to the Amazon S3 bucket. Change the permissions on these two files so that they are publicly accessible.
  3. Enable the Amazon S3 bucket for static website hosting. Note down the endpoint address shown in the Static website hosting section.This is the website’s address (URL).

Thats it folks! The Life Insurance Bot will now be alive!. To access it, open your favourite internet browser and go to the static website’s endpoint address.

The final product!

I must admit, a lot of work went into making this Amazon Lex Bot, however it is easily justified by the end result! One thing I would like to state is that, this prototype didn’t take more than 2 days to build. The speed at which you can create proof-of-concepts in AWS gives you a great advantage over your competitors.

Below is a screenshot of what my LifeInsuranceBot looks like in action. If you followed through, yours would be similar.

I hope this blog was useful and comes in handy when you are trying to create your own web chatbots.

Till the next time, Enjoy!

Display Control Plane API Operations using Amazon CloudWatch Logs Insights

Introduction

For small organisations that cannot afford to spend much on their network security, moving to the cloud enables them to easily uplift their security posture. The organisation can concentrate on innovating and scaling their workloads, while AWS provides them with a secure environment to use. More information regarding AWS security and compliance can be found at https://docs.aws.amazon.com/whitepapers/latest/aws-overview/security-and-compliance.html

The sentence above is not entirely accurate. The security of workloads in AWS is a shared responsibility between AWS and the customer. AWS is responsible for “Security of the cloud”. This means that AWS is responsible for protecting the infrastructure that runs the workloads (hardware, software, networking, physical facilities). The customer is responsible for “Security in the cloud”. This means that the customer, based on the service they select, must perform all the necessary security configuration and management tasks to keep the workload secure. A good place to learn more about this is at https://aws.amazon.com/compliance/shared-responsibility-model/

As a good security practice, one must always monitor all the activities that happen in their cloud environment, especially those that involve the management of resources. For instance, if one notices a lot of large Amazon EC2 instances being provisioned, this could possibly be an indication of a breach (or someone authorised is provisioning these without notifying others). The management operations performed on resources in your AWS account are referred to as Control Plane operations.

There are many commercial products that can help you with monitoring your AWS environment. However, you can quite adequately benefit from the tools that are natively provided by AWS as well.

In this blog, I will take you through the steps of using Amazon CloudWatch Logs Insights, to easily display the Control Plane operations, in a meaningful way.

Implementation

Let’s start:

  1. Open the AWS Console and then navigate to the AWS CloudTrail service page. Change to the appropriate AWS region.
  2. Create a new Trail to record just Management events. Ensure it is applied to all the regions and that it delivers events to Amazon CloudWatch Logs as well. Below is a screenshot of the required settings

  3. Open the Amazon CloudWatch service page and ensure it is in the same region as the AWS CloudTrail that was just configured (in step 2 above)
  4. Click on Dashboards and then click on Create dashboard. Give the dashboard a meaningful name (I called my dashboard ControlPlaneOperations-Dashboard)
  5. In the next screen, from the top menu, click on Add widget. Another screen will open. Select Query results and then click on Configure

  6. In the next screen, use the drop-down arrow (pointed by the red arrow below) to select the CloudWatch Log group that was configured for the new CloudTrail in step 1 above.

    In the formula section (denoted by the red rectangle), delete everything and replace it with the text below (the screenshot above already has the correct formula)

    fields eventTime, userIdentity.userName, userIdentity.accessKeyId, sourceIPAddress, awsRegion, eventSource, eventName | sort eventTime desc
    

    The above formula directs Amazon CloudWatch Log Insights to display the event time, user name and access key of the identity that performed the control plane operation, the ip address from where the operation was performed, the AWS region inside which the operation was performed, the event’s source and name. The results are sorted based on the event time in descending order.

    Next, set the time range for events that CloudWatch Logs Insights must process. To configure this, pick the appropriate duration from those displayed on the top right (as displayed inside the green rectangle in the screenshot above). For my setup, I chose 1hr.

    Once completed, click Create widget

  7. The next screen should look similar the screenshot below. Click Save dashboard

     

    Amazon CloudWatch Logs Insights - Create Dashboard

  8. Your Amazon CloudWatch dashboard is now complete. To refresh the events, you can press the refresh button (pointed by the red arrow in the screenshot
  9. Amazon CloudWatch Logs Insights - Refresh Events
  10. You can also enable auto refresh of the events by clicking the small arrow beside the refresh button. You will get a menu option similar to the screenshot below.

    Tick Auto refresh and choose the Refresh interval of your choice.

  11. Pro Tip 1 – if you want your dashboard to be displayed under the Favorite section when you open the Amazon CloudWatch service page, go into the Dashboards section of Amazon CloudWatch and click on Favorite (star) beside your dashboard name.
  12. Pro Tip 2 – If you want your dashboard to appear on the default Amazon CloudWatch service page, rename your dashboard to CloudWatch-Default.
  13. Pro Tip 3 = at the beginning of each row In your dashboard, you will notice a small arrow head. If you click on the arrow head, it expands that event and provides additional information.

Thats it! You should now have a dashboard similar to the one below that shows the control plane operations as they happen.

Amazon CloudWatch Logs Insights Dashboard

At times, I found there to be approximately five minutes of delay between when the event happened and when it was displayed. This could be due to the delay between when the event was generated and when that service that generated it delivered the logs to AWS CloudTrail.

The dashboard should allow you to easily monitor any suspicious control plane activities in your AWS account.

I hope the above was useful. Till the next time, Enjoy!

Creating a Contact Center in minutes using Amazon Connect

Background

In my previous blog (https://nivleshc.wordpress.com/2019/10/09/managing-amazon-ec2-instances-using-amazon-ses/), I showed how we can manage Amazon EC2 instances using emails. However, what if you wanted to go further than that? What if, instead of sending an email, you instead wanted to dial in and check the status of or start/stop your Amazon EC2 instances?

In this blog, I will show how I used the above as a foundation to create my own Contact Center. I enriched the experience by including an additional option for the caller, to be transferred to a human agent. All this in minutes! Still skeptical? Follow on and I will show you how I did all of this using Amazon Connect.

High Level Solution Design

Below is the high-level solution design for the Contact Center I built.

The steps (as denoted by the numbers in the diagram above) are explained below

  1. The caller dials the Direct Inward Dial (DID) number associated with the Amazon Connect instance
  2. Amazon Connect answers the call
  3. Amazon Connect invokes the AWS Lambda function to authenticate the caller.
  4. The AWS Lambda function authenticates the caller by checking their callerID against the entries stored in the authorisedCallers DynamoDB table. If there is a match, the first name and last name stored against the callerID is returned to Amazon Connect. Otherwise, an “unauthorised user” message is returned to Amazon Connect.
  5. If the caller is unauthorised, Amazon Connect informs them of this and hangs up the call.
  6. If the caller is authorised, Amazon Connect uses the first name and last name provided by AWS Lambda function to personalise a welcome message for them. Amazon Connect then provides the caller with two options:
      •  (6a) press 1 to get the status of the Amazon EC2 instances. If this is pressed, Amazon Connect invokes an AWS Lambda function to get the status of the Amazon EC2 instances and plays the results to the caller
      • (6b) press 2 to talk to an agent. If this is pressed, Amazon Connect places the call in a queue,  where it will be answered by the next available agent

     

Preparation

My solution requires the following components

  • Amazon DynamoDB table to store authorised callers (an item in this table will have the format phonenumber, firstname,  lastname)
  • AWS Lambda function to authenticate callers
  • AWS Lambda function to get the status of all Amazon EC2 instances in the region

I created the following AWS CloudFormation template to provision the above resources.

AWSTemplateFormatVersion: "2010-09-09"
Description: Template for deploying Amazon DynamoDB and AWS Lambda functions that will be used by the Amazon Connect instance
Parameters:
authorisedUsersTablename:
Description: Name of the Amazon DynamoDB Table that will be created to store phone numbers for approved callers to Amazon Connect
Type: String
Default: amzn-connect-authorisedUsers
DynamoDBBillingMode:
Description: Billing mode to be used for authorisedUsers Amazon DynamoDB Table
Type: String
AllowedValues: [PAY_PER_REQUEST]
Resources:
authoriseCallerLambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Principal:
Service:
lambda.amazonaws.com
Action:
sts:AssumeRole
Path: "/"
Policies:
PolicyName: logsStreamAccess
PolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Action:
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
PolicyName: DynamoDBAccess
PolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Action:
dynamodb:Query
Resource: !GetAtt authorisedUsersTable.Arn
getInstanceStatusLambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Principal:
Service:
lambda.amazonaws.com
Action:
sts:AssumeRole
Path: "/"
Policies:
PolicyName: logsStreamAccess
PolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Action:
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
PolicyName: EC2DescribeAccess
PolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Action:
"ec2:Describe*"
Resource: "*"
authoriseCallerFunctionPolicy:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt
authoriseCaller
Arn
Principal: connect.amazonaws.com
getInstanceStatusFunctionPolicy:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt
getInstanceStatus
Arn
Principal: connect.amazonaws.com
authorisedUsersTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: !Ref authorisedUsersTablename
AttributeDefinitions:
AttributeName: phoneNumber
AttributeType: S
KeySchema:
AttributeName: phoneNumber
KeyType: HASH
BillingMode: !Ref DynamoDBBillingMode
authoriseCaller:
Type: AWS::Lambda::Function
Properties:
FunctionName: "amzn-connect-authoriseCaller"
Description: "This function checks if the caller is authorised to use the Amazon Connect Service"
Handler: index.lambda_handler
Runtime: python3.6
Role: !GetAtt 'authoriseCallerLambdaExecutionRole.Arn'
Environment:
Variables:
AUTHORISEDUSERSTABLE: !Ref authorisedUsersTable
Code:
ZipFile: |
import boto3
import os
from boto3.dynamodb.conditions import Key, Attr
def lambda_handler(event, context):
print("event:",event)
print("context:",context)
authorisedUsersTable = os.environ['AUTHORISEDUSERSTABLE']
callerID = event["Details"]["ContactData"]["CustomerEndpoint"]["Address"]
#Establish connection to dynamoDB and retrieve table
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(authorisedUsersTable)
response = table.query(KeyConditionExpression=Key('phoneNumber').eq(callerID))
if (len(response['Items']) > 0):
firstName = response['Items'][0]['firstName']
lastName = response['Items'][0]['lastName']
else:
firstName = 'unauthorised'
lastName = 'unauthorised'
callerDetails = {
'phoneNumber' : callerID,
'firstName' : firstName,
'lastName' : lastName
}
print("CallerDetails:",str(callerDetails))
return callerDetails
getInstanceStatus:
Type: AWS::Lambda::Function
Properties:
FunctionName: "amzn-connect-getInstanceStatus"
Description: "This function checks and reports the number of EC2 instances that are running and stopped in the AWS region where this AWS Lambda function is running"
Handler: index.lambda_handler
Runtime: python3.6
Role: !GetAtt 'getInstanceStatusLambdaExecutionRole.Arn'
Code:
ZipFile: |
import boto3
def lambda_handler(event, context):
print("event:",event)
print("context",context)
ec2 = boto3.client("ec2")
ec2_status_running = ec2.describe_instances(
Filters=[
{
'Name':'instance-state-name',
'Values':['running']
}
]
)
ec2_status_running = ec2.describe_instances(
Filters=[
{
'Name':'instance-state-name',
'Values':['running']
}
]
)
ec2_status_stopped = ec2.describe_instances(
Filters=[
{
'Name':'instance-state-name',
'Values':['stopped']
}
]
)
num_ec2_running = len(ec2_status_running['Reservations'])
num_ec2_stopped = len(ec2_status_stopped['Reservations'])
result = {
'numberEC2Running': num_ec2_running,
'numberEC2Stopped': num_ec2_stopped
}
print("Number of EC2 instances running:",num_ec2_running)
print("Number of EC2 instances stopped:",num_ec2_stopped)
return result

The above AWS CloudFormation template can be downloaded from https://gist.github.com/nivleshc/926259dbbab22dd4890e0708cf488983

Implementation

Currently AWS CloudFormation does not support Amazon Connect. The implementation must be done manually.

Leveraging on my own experience with setting up Amazon Connect solutions,  I observed that there are approximately three stages that are required to get a Contact Center up and running. These are:

  • Provisioning an Amazon Connect instance – this is straight forward and essentially is where an Amazon Connect instance is provisioned and made ready for your use
  • Configuring the Amazon Connect instance – this contains all the tasks to customise the Amazon Connect instance. It includes the configuration of the Direct Inward Dial (DID), hours or operations for the Contact Center, Routing profiles, users etc
  • Creating a custom Contact flow – a Contact flow defines the customer experience of your Contact Center, from start to finish. Amazon Connect provides a few default Contact flows however it is highly recommended that you create one that aligns with your own business requirements.

Follow along and I will show you how to go about setting up each of the above mentioned stages.

Provision the Amazon Connect Instance

  1. From the AWS Console, open the Amazon Connect service. Select the Sydney region (or a region of your choice, however do keep in mind that at the moment, Amazon Connect is only available in a few regions)
  2. Enter an Access URL for your Amazon Connect Instance. This URL will be used to access the Amazon Connect instance once it has been provisioned.
  3. In the next screen, create an administrator account for this Amazon Connect instance
  4. The next prompt is for Telephony options. For my solution, I selected the following:
    1. Incoming calls: I want to handle incoming calls with Amazon Connect
    2. Outgoing calls: I want to make outbound calls with Amazon Connect
  5. In the next screen, Data Storage options are displayed. For my solution, I left everything as default.
  6. In the next screen, review the configuration and then click Create instance

Configure the Amazon Connect Instance

After the Amazon Connect instance has been successfully provisioned, use the following steps to configure it:

  1. Claim a phone number for your Amazon Connect Instance. This is the number that users will be calling to interact with your Amazon Connect instance (for claiming non toll free local numbers, you must open a support case with AWS, to prove that you have a local business in the country where you are trying to claim the phone number. Claiming a local toll-free number is easier however it is more expensive)
  2. Create some Hour of operation profiles. These will be used when creating a queue
  3. Create a queue. Each queue can have different hours of operation assigned
  4. Create a routing profile. A user is associated with a routing profile, which defines their inbound and outbound queues.
  5. Create users. Once created, assign the users to a predefined security profile (administrator, agent etc) and also assign them to a specific routing profile

Create a custom Contact flow

A Contact flow defines the customer experience of your Contact Center, from start to finish. By default, Amazon Connect provides a few Contact flows that you can use. However, it is highly recommended that you create one that suits your own business requirements.

To create a new Contact flow, follow these steps:

  • Login to your Amazon Connect instance using the Access URL and administrator account (you can also access your Amazon Connect instance using the AWS Console and then click on Login as administrator)
  • Once logged in, from the left-hand side menu, go to Routing and then click on Contact flows
  • In the next screen, click on Create contact flow
  • Use the visual editor to create your Contact flow

Once the Contact flow has been created, attach it to your Direct Inward Dial (DID) phone number by using the following steps:

  • from the left-hand side menu, click on Routing and then Phone numbers.
  • Click on the respective phone number and change its Contact flow / IVR to the Contact flow you want to attach to this phone number.

Below is a screenshot of the Contact flow I created for my solution. It shows the flow logic I used and you can easily replicate it for your own environment. The red rectangles show where the AWS Lambda functions (mentioned in the pre-requisites above) are used.

This is pretty much all that is required to get your Contact Center up and running. It took me approximately thirty minutes from start to finish (this does not include the time required to provision the Amazon DynamoDB tables and AWS Lambda functions). However, I would recommend spending time on your Contact flows as this is brains of the operation. This must be done in conjunction with someone who understands the business really well and knows the outcomes that must be achieved by the Contact Center solution. There is a lot that can be done here and the more time you invest in your Contact flow, the better outcomes you will get.

The above is just a small part of what Amazon Connect is capable of. For its full set of capabilities, refer to https://aws.amazon.com/connect/

So, if you have been dreaming of building your own Contact Center, however were worried about the cost or effort required? Wait no more! You can now easily create one in minutes using Amazon Connect and pay for only what you use and tear it down if you don’t need it anymore. However, before you start, I would strongly recommend that you get yourself familiar with the Amazon Connect pricing model. For example – you get charged a daily rate for any claimed phone numbers that are attached to your Amazon Connect Instance (this is similar to a phone line rental charge). Full pricing is available at https://aws.amazon.com/connect/pricing/).

I hope the above has given you some insights into Amazon Connect. Till the next time, Enjoy!

Managing Amazon EC2 Instances using Amazon SES

Background

Most people know Amazon Simple Email Service (SES) just as a service for sending out emails. However, did you know that you can use it to receive emails as well? If this interests you, more information is available at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email.html.

In this blog I will show how I provisioned a solution to manage my Amazon EC2 instances using emails. The solution uses Amazon SES and AWS Lambda. Now, some of you might be saying, can’t you just use the AWS console or app for this? Well, yes you can, however for me personally, logging into an AWS console just to get the status of my Amazon EC2 instances, or to start/stop them was more effort than I deemed necessary. The app surely makes this task trivial, however the main purpose of this blog is to showcase the capabilities of Amazon SES

Solution Architecture

Below is the high-level design for my solution.

The individual steps (labelled using numbers) are described below

  1. The admin sends an email to an address attached to an Amazon SES rule
  2. Amazon SES receives the email, performs a spam and virus check. If the email passes the check, Amazon SES invokes the manageInstances AWS Lambda function, passing the contents of the email to it (unfortunately the contents of the body are not passed)
  3. The manageInstances AWS Lambda function authenticates the sender based on the from address (this is a very rudimentary authentication system. A stronger authentication mechanism must be used if this solution is to be deployed in a production environment – maybe include a multi-factor authentication system). It extracts the command from the email’s subject and executes it
  4. The manageInstances AWS Lambda function uses Amazon SES to send the response of the command back to the admin
  5. Amazon SES delivers the email containing the command’s output to the admin

Prerequisites

To use Amazon SES for receiving incoming emails, first verify your domain within it and then point your domain’s DNS MX entry to your region’s Amazon SES endpoint. Full Instructions to carry this out can be found at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-getting-started.html

Implementation

Lambda Function

The Lambda function is created first as it is required for the Amazon SES Rules.

The Lambda function carries out the following tasks

  • authenticates the sender by comparing the email’s from address with a predefined list of approved senders. To prevent a situation where the Lambda function can be inadvertently used as a spam bot, all emails from senders not on the approved list will be dropped.
  • checks if the specified command is in the list of commands that is currently supported. If yes, then the command is executed, and the output sent back to the admin. If the command is unsupported, a reply stating that an invalid command was specified is sent back to the admin

Here are the attributes of the Lambda function I created

Function name: manageInstances
Runtime:             Python 3.6
Region:                North Virginia (us-east-1) This is to ensure that the Lambda function and the Amazon SES rules are in the same region
Role:                     The role that is used by the Lambda function must have the following permissions attached to it

             ec2:DescribeInstances
             ec2:StartInstances
             ec2:StopInstances
             SES:SendEmail

Here is the code for the AWS Lambda function (set the approvedSenders list to contain the email address of approved senders)

import boto3
def provideHelp(params):
message = "Set the subject of the email to one of the following\n"
message += "help – provides this help\n"
message += "status – provides the status of all ec2 instances in the region\n"
message += "start {instance-id} – starts the ec2 instance with with the specified instance-id\n"
message += "stop {instance-id} – stops the ec2 instance with the specified instance id"
return message
def getStatus(params):
#get a list of all ec2 instances
print("getStatus:Params:",params)
#if a parameter is provided, it is the aws region to check
if (len(params) >= 1):
ec2 = boto3.client("ec2", params[0])
else:
ec2 = boto3.client("ec2")
response = ec2.describe_instances()
instances = response['Reservations']
number_instances = len(instances)
print("NumInstances:",number_instances)
message = ""
for instance in instances:
print("Instance:",str(instance))
instance_id = instance['Instances'][0]['InstanceId']
message += instance_id + "\t"
try:
if (instance['Instances'][0]['Tags'][0]['Key'] == 'Name'):
instance_name = instance['Instances'][0]['Tags'][0]['Value']
if (instance_name == ""):
message += "[No Name Found]\t"
else:
message += instance_name + "\t"
except:
message += "[No Name Found]\t"
instance_state = instance['Instances'][0]['State']['Name']
message += instance_state + "\t"
try:
instance_privateip = instance['Instances'][0]['PrivateIpAddress']
message += instance_privateip + "\t"
except:
message += "[No Private IP]\t"
try:
instance_publicip = instance['Instances'][0]['PublicIpAddress']
message += instance_publicip + "\t"
except:
message += "[No Public IP]\t"
message += "\n"
return message
def startInstance(params):
if (len(params) > 1):
instanceId = params[0]
aws_region = params[1]
ec2 = boto3.client("ec2", aws_region)
else:
instanceId = params[0]
ec2 = boto3.client("ec2")
try:
response = ec2.start_instances(
InstanceIds=[instanceId]
)
message = "Starting instance " + str(instanceId) + " Please check status in a couple of minutes\n" + str(response)
except Exception as e:
message = "Error starting instance(s) :" + str(instanceId) + " " + str(e)
print(message)
return message
def stopInstance(params):
if (len(params) > 1):
instanceId = params[0]
aws_region = params[1]
ec2 = boto3.client("ec2", aws_region)
else:
instanceId = params[0]
ec2 = boto3.client("ec2")
try:
response = ec2.stop_instances(
InstanceIds=[instanceId]
)
message = "Stopping instance " + str(instanceId) + " Please check status in a couple of minutes\n" + str(response)
except Exception as e:
message = "Error stopping instance(s) " + str(instanceId) + " " + str(e)
print(message)
return message
def sendEmail(fromAddress,recipientAddress,subject,body):
#sending email
print("sending email")
client = boto3.client('ses')
response = client.send_email(
Destination={
'ToAddresses': [recipientAddress],
},
Message={
'Body': {
'Text': {
'Charset': 'UTF-8',
'Data': body
},
},
'Subject': {
'Charset': 'UTF-8',
'Data': subject
},
},
ReplyToAddresses=[fromAddress],
ReturnPath=fromAddress,
Source=fromAddress
)
print("Response from SES:",response)
def lambda_handler(event, context):
approvedSenders = ['john@example.com','tom@example.com','jane@example']
print("Event:",event)
print("Context",context)
emailSender = event["Records"][0]["ses"]["mail"]["source"]
emailSubject = event["Records"][0]["ses"]["mail"]["commonHeaders"]["subject"]
command_split = emailSubject.split(" ")
command = command_split[0].lower()
if (len(command_split) > 1):
commandParams = command_split[1:len(command_split)]
else:
commandParams = "" #if there are no command params, then just set it to blank
print("From:",emailSender)
print("Subject:",emailSubject)
print("Command:",command)
#authenticate the sender based on fromAddress
if emailSender in approvedSenders:
switcher = {
"help": provideHelp,
"status": getStatus,
"start": startInstance,
"stop": stopInstance
}
#get the command that was specified in the email
functionToRun = switcher.get(command, lambda params: "invalid command")
body = functionToRun(commandParams)
sendEmail("admin@managedinstances.com",emailSender,"Execution result for command:"+ emailSubject, str(body))
else:
print("Sender ",emailSender," not approved for executing commands. Ignore")

view raw
manageInstances.py
hosted with ❤ by GitHub

The AWS Lambda function code can be downloaded from https://gist.github.com/nivleshc/f9b32a14d9e662701c3abcbb8f264306

Amazon SES Email Receiving Rule

Next, the Amazon SES Email Receiving rule that handles the incoming emails must be created. Please note that currently Amazon SES is supported in a few regions only. For my solution, I used the North Virginia (us-east-1) region.

Below are the steps to create the rule

  • Open Simple Email Service service page from within the AWS console (ensure you are in the correct AWS region)
  • From the left-hand side menu, navigate down to the Email Receiving section and then click Rule Sets.
  • The right-hand side of the screen will show the currently defined rule sets. I used the predefined default-rule-set.
  • Click on View Active Rule Set and then in the next screen click Create Rule.
  • In the next screen, for the recipient address, enter the email address to which emails will be sent to, to carry out the commands (the email domain has to correspond to the domain that was verified with Amazon SES, as part of the prerequisites mentioned above)
  • In the next screen, for actions, select Lambda.
  • Select the name of the Lambda function that was created to manage the instances from the drop down (for me, this was manageInstances)
  • Ensure the Invocation type is set to Event.
  • You do not need to set the SNS topic, however if you need to know when this Amazon SES action is carried out, select the appropriate SNS topic (you will need to create an SNS topic and subscribe to it using your email address)
  • Click Next Step.
  • In the next screen, provide a name for the rule. Ensure the options Enabled and Enable spam and virus scanning are ticked.
  • Click Next Step and then review the settings.
  • Click Create Rule.

Usage

The solution, once implemented supports the following commands

help   - provides information about the commands and their syntax
status - provides the status of all Amazon EC2 instances in the region that the Lambda function is running in. It lists the instance-id and name of those instances (name is derived from the tag with the key Name)
start {instance-id} - starts the Amazon EC2 instance that has the specified instance-id
stop {instance-id}  - stops the Amazon EC2 instance that has the specified instance-id

To use, send an email from an approved sender’s email to the email address attached to Amazon SES.

The table below shows, what the subject must be, for each command.

Command Subject
Help help
Status for instances in us-east-1 status
Start an instance with instance-id i-0e7e011b42e814465 start i-0e7e011b42e814465
Stop an instance with instance-id i-0e7e011b42e814465 stop i-0e7e011b42e814465

The output of the status command is in the following format

<instance-id> <instance-name> <status> <private-ip> <public-ip>

for example
i-03a1ab124f554z805 LinuxServer01 Running 172.16.31.10 52.10.100.34

The only problem with the solution is that all commands are performed on Amazon EC2 instances running in the same AWS region as the Lambda function. What if you wanted to carry out the commands on another region?

For the keen eyed, you would have spotted the Easter egg I hid in the Lambda function code. Here is what the subject must be if the command is to be carried out in an AWS region different to where the Lambda function is running (simply provide the AWS region at the end of the command)

Command Subject
Help help
Status for instances in ap-southeast-2 (Sydney) status ap-southeast-2
Start an instance with instance-id i-0e7e011b42e814465 in ap-southeast-2 (Sydney) region start i-0e7e011b42e814465 ap-southeast-2
Stop an instance with instance-id i-0e7e011b42e814465 in ap-southeast-2 (Sydney) region stop i-0e7e011b42e814465 ap-southeast-2

There you go! Now you can keep an eye on and control your Amazon EC2 instances with just your email.

A good use case can be when you are commuting and need to RDP into your Windows Amazon EC2 instance from your mobile (I am guilty of doing this at times). You can quickly start the Amazon EC2 instance, get its public ip address, and then connect using RDP.  Once finished, you can shut down the instance to ensure you don’t get charged after that.

I hope this blog was useful to you. Till the next time, Enjoy!

Using Serverless Framework and AWS to map Near-Realtime Positions of Trains

Background

A couple of months back, I found out about the Open Data initiative from Transport for New South Wales. This is an awesome undertaking, to provide data to developers and other interested parties, so that they can develop great applications. For those interested, the Open Data hub can be accessed at https://opendata.transport.nsw.gov.au.

I had been playing with data for a few months now and when I looked through the various APIs that I could access via the Open Data hub, I became extremely interested.

In this blog, I will take you through one of my mini projects based off the data at Open Data Hub. I will be using the Public Transport – Vehicle Positions API to plot the near realtime positions of Sydney trains on a map. The API provides access to more than just train position data, however to keep things simple, I will concentrate on only trains in this blog.

Solution Architecture

As I am a huge fan of serverless, I decided to architect my solution with as much serverless components as possible. The diagram below shows a high-level architecture of how the Transport Positioning System (this is what I will call my solution, TPS for short) will be created.

Let’s go through the steps (as marked in the diagram above)

  1. The lambda function will query the Open Data API every 5 minutes for the position of all trains
  2. After the data has been received, the lambda function will go through each record and assign a label to each train. To ensure the labels are consistent across each lambda invocation, the train to label association will be stored in an Amazon DynamoDB table. The lambda function will query the table to check if a train has already been allocated a label. If it has, then that label will be used. Otherwise, a new label will be created, and the Amazon DynamoDB table will be updated to store this new train to label association.
  3. I found Bing Maps to be much easier (and cheaper) to use for plotting items on a map. The only disadvantage is that it can, at most show 100 points (called pushpins) on the map. The lambda function will go through the first 100 items returned from Open Data API and using the labels that were found/created in step 2, create a pushpin url that will be used to generate a map of the location of the first 100 trains. This url will then be used to generate the map by sending a request to Bing Maps.
  4. The lambda function will then create a static webpage that displays the map showing the position of the trains, along with a description that provides more information about the labels used for each train (for example, label 1 could have a description of “19:10 Central Station to Penrith Station”). The label description is set to the train’s label, which is obtained from the query results of the Open Data API query.

This is what I cooked up earlier

Be warned! This blog is quite lengthy as a lot was done to make this solution work. However, if you would rather see the final result before delving into the details, check it out https://sls-tps-website-dev.s3-ap-southeast-2.amazonaws.com/vehiclelocation.html.

Screenshot of TPS

Above is a screenshot of TPS in action. It is a static webpage that is being generated every 5 minutes, showing the position of trains. I will keep the lambda function running for at least three months, so you have plenty of time available to check it out.

Okay, let’s get our hands dirty and start coding.

Prerequisites

Before we start, the following must be in place

  1. You must have an AWS account. If you don’t have one already, you can sign up for a free tier at https://aws.amazon.com/free/
  2. You must have Serverless Framework installed on your computer. If you don’t have, it follow the details at https://serverless.com/framework/docs/getting-started/
  3. Setup the AWS access keys and secret access key that Serverless Framework will use to provision resources into your AWS account. Instructions to get this done can be obtained from https://serverless.com/framework/docs/providers/aws/guide/credentials/
  4. After items 1 – 3 have been completed, create a python runtime Serverless Framework service (my service is called tps)
  5. Within the tps service folder, install the following Serverless Framework plugins
    1. serverless-python-requirements – this plugin adds all the required python modules into a zip file containing our lambda function script, which then gets uploaded to AWS (the required python modules must be defined in the requirement.txt file)
    2. serverless-prune-plugin – this plugin ensures that only the specified number of lambda function versions exist.
  6. Serverless-python-requirements requires a file called requirements.txt. For this solution, create a file called requirements.txt at the root of the service folder and put the following lines inside it
    requests==2.22.0
    gtfs-realtime-bindings==0.0.6
  7. Register an account with Transport for New South Wales Open Data Hub (https://opendata.transport.nsw.gov.au/). This is free. Once registered, login to the Open Data Hub portal and then under My Account, click on Applications and then create an Application that has permissions to Public Transport – Realtime Vehicle Positions API. Note down the API key that is generated as it will be used by the python script later.
  8. Create an account with Bing Maps, use the Website licence plan. This will provide 125,000 billable transactions of generating maps per calendar year at no charge. For generating a map every 5 minutes, this is more than enough. Note down the API key that is provided. Details on creating a Bing Maps account is available at https://www.microsoft.com/en-us/maps/licensing/options

This project has two parts to it. The first is to create the AWS resources that will host our project. For this, we will be using Serverless Framework to create our AWS Lambda function, Amazon Simple Storage Service (S3) bucket, Amazon DynamoDB table, AWS CloudWatch Logs and AWS CloudWatch Events.

The second part is the API query and data ingestion from the Open Data API. The next sections will cover each of these parts.

AWS Resource Creation

As previously mentioned, we will use Serverless Framework to create our AWS resources. Serverless Framework uses serverless.yml to specify which resources need to be created. This file is created by default whenever a Serverless Framework service is created.

In this section, I will take you through the serverless.yml file I used for this project.

The file starts like this.

service: sls-${self:custom.application}
plugins:
serverless-python-requirements
serverless-prune-plugin
custom:
pythonRequirements:
dockerizePip: true
slim: true
noDeploy:
typing
stage: ${opt:stage, self:custom.defaults.stage}
region: ${opt:region, self:custom.defaults.region}
application: ${env:APPLICATION, self:custom.defaults.application}
logRetentionInDays: ${opt:logretentionindays, env:logretentionindays, self:custom.defaults.logretentionindays}
s3:
websiteBucket: sls-${self:custom.application}-website-${self:custom.stage}
dynamodb:
transportPositionTableName: transportPosition
billingMode: PAY_PER_REQUEST
defaults:
application: tps
stage: dev
region: ap-southeast-2
logretentionindays: 14
prune:
automatic: true
number: 1

It defines the service name, plugins and variables that will be used in this file (notice the plugins serverless-python-requirements and serverless-prune-plugin)

The following default values have been configured in the above serverless.yml file.

  • application name is set to tps
  • environment is set to dev
  • Amazon CloudWatch logs is set to 14 days retention
  • the AWS region is set to ap-southeast-2 (Sydney)
  • the Amazon DynamoDB table is set to transportPosition
  • the billing mode for the Amazon DynamoDB table will be set to PAY PER REQUEST.

The next section defines the details for the cloud provider where resources will be provisioned.

provider:
name: aws
runtime: python3.7
endpointType: regional
stage: ${self:custom.stage}
region: ${self:custom.region}
memorySize: 256
timeout: 300
versionFunctions: false
deploymentBucket: sls-${self:custom.application}-deploymentbucket
logRetentionInDays: ${self:custom.logRetentionInDays}
environment:
STAGE: ${self:custom.stage}
REGION: ${self:custom.region}

As previously mentioned, I am using AWS. The lambda function will use python 3.7 runtime. The deployment bucket to host all artefacts is also defined. This is an Amazon Simple Storage Service (S3) bucket. Ensure that this S3 bucket exists before deploying the serverless service.

The next section defines the IAM role that will be created for the lambda function.

# define IAM roles
iamRoleStatements:
Effect: Allow
Action:
s3:HeadObject
s3:GetObject
s3:GetObjectAcl
s3:PutObject
s3:PutObjectAcl
s3:CreateMultiPartUpload
Resource:
arn:aws:s3:::${self:custom.s3.websiteBucket}/*
Effect: Allow
Action:
dynamodb:*
Resource:
!GetAtt transportPositionTable.Arn
!Join
''
– !GetAtt transportPositionTable.Arn
'/index/TripDate-GSI'

The specified IAM role will allow the lambda function to carry out all required operations on Amazon Dynamodb table (currently the IAM role permits all DynamoDB actions, however if required, this can be tightened) and to upload objects to the Amazon S3 bucket that will host the static website.

The next section provides instructions on what to include and exclude when creating the serverless package.

# you can add packaging information here
package:
include:
src/csv/*.csv
exclude:
__cache__/**
__pycache__/**
events/**
node_modules/**
'conf/**/*'
'gulp-libs/**/*'
'yarn.lock'
'package.json'
'.dotenv'

Now we come to the important sections for within serverless.yml. The next section defines the lambda function, its handler and what events will trigger it.

functions:
tpsFindVehiclePositionsCron:
handler: src.tps_vehiclepos.run
memorySize: 3008
timeout: 900
environment:
BUCKET: ${self:custom.s3.websiteBucket}
TRANSPORTPOSITION_TABLE: ${self:custom.dynamodb.transportPositionTableName}
events:
schedule:
rate: cron(0/5 * * * ? *)
enabled: true

For the tps lambda function, the handler function is at src.tps_vehiclepos.run (this means that there is a subfolder within the service folder called src, within which there is a python file called tps_vehiclepos. Inside this file is a function called run). The lambda function will be run every 5 minutes. To achieve this, I am using AWS Events. Two environment variables are also being passed to the lambda function (BUCKET and TRANSPORTPOSITION_TABLE)

The last section defines all the resources that will be created by the Serverless Framework.

resources:
Description: Transport Positioning System Resources
Resources:
websiteBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:custom.s3.websiteBucket}
#DynamoDB will be used to store the label that each transport will be given.
#The label will be used as the pushpin label on the map.
#DynamoDB will have attributes TripDate, TripId, VehicleId, ExpirationTime(TTL),
#TimeAdded, Latitude, Longitude, PushpinLabel, PushPinLabelDescr
transportPositionTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${self:custom.dynamodb.transportPositionTableName}
AttributeDefinitions:
AttributeName: TripDate
AttributeType: S
AttributeName: VehicleId
AttributeType: S
KeySchema:
AttributeName: TripDate
KeyType: HASH
AttributeName: VehicleId
KeyType: RANGE
BillingMode: ${self:custom.dynamodb.billingMode}
GlobalSecondaryIndexes:
IndexName: TripDate-GSI
KeySchema:
AttributeName: TripDate
KeyType: HASH
Projection:
ProjectionType: ALL
TimeToLiveSpecification:
AttributeName: ExpirationTime
Enabled: true

The following resources will be created

  • Amazon S3 bucket. This bucket will store the Bing Maps that show the train positions. It will also serve the website which will be used to display the position maps.
  • Amazon DynamoDB table. The table will be used to store details for trains found in the Open Data API query. Note that we are also using the DynamoDB TTL feature. Since we don’t need to retain items older than a day, this will allow us to easily prune the Amazon DynamoDB table, to reduce costs.

The full serverless.yml file can be downloaded from https://gist.github.com/nivleshc/951ff2f235a89d58d4abec6de91ef738

Generating the map

In this section we will go through the script that does all the magic, which is querying the Open Data API, processing the data, generating the map and then displaying the results in a webpage.

The script is called tps_vehiclelocation.py. I will discuss the script in parts below.

The first section lists the python modules that will be imported. It also defines the variables that will be used within the script.

#this script will query OpenData API to get the current positions for Sydney Trains. It will then use this information to plot the
#positions on a map
from google.transit import gtfs_realtime_pb2
import requests
import datetime
import boto3
import os
import time
from pytz import timezone
from decimal import Decimal
#list all variables –start
baseURL = 'https://api.transport.nsw.gov.au/v1/gtfs/vehiclepos/&#39; # Define URL for opendata API
headers = {'Authorization': 'apikey <insert your opendata api key here>'} # Define header for opendata API
bucketName = os.environ['BUCKET'] # Set up bucket from environment variable
transportPositionTable = os.environ['TRANSPORTPOSITION_TABLE'] #dynamodb table for pushpin labels
bucketImagesKey = "images/" # set bucket key for images
HtmlLandingpagename = "vehiclelocation.html" #this is the landing html page that will load the image file to show the current position of vehicles
#Define operators for opendata API. Possible operators are ['sydneytrains', 'buses', 'ferries', 'lightrail', 'nswtrains', 'regionbuses', 'metro']
operators = ['sydneytrains']
#we will use bing maps to plot the locations
maps_bing_prefix = "https://dev.virtualearth.net/REST/v1/Imagery/Map/Road?mapsize=20000,20000&&quot;
maps_bing_apikey = "format=jpeg&dcl=1&key=<insert your bing maps api key here>"
maps_error_imageURL = "https://image.shutterstock.com/image-photo/image-white-text-message-will-600w-1012395526.jpg&quot; #show this image when there are errors
#generating map
maps_bing_pushpin_limit = 100 #this is the maximum pushpins that bing maps allows
inactive_vehicle_datetime_limit = 10 #(in minutes) this will be used to identify inactive vehicles. If a vehicle's
#last stop id timestamp is at least inactive_vehicle_datetime_limit minutes old,
# then it is considered to be inactive
maps_active_vehicle_icon = '4' #this is the icon that will be used to show active vehicles on the map
maps_inactive_vehicle_icon = '10' #this is the icon that will be used to show inactive vehicles on the map
legend_vehicle_displayed_on_map_color = "#00bfff" #this color will be used to show legend entries for which the vehicle is displayed on map
lastpushpinLabel = 0 #this will be used to generate labels for pushpins. Each pushpin will have a number, which will be associated with a description.
#the association will be displayed under a legends section on the right hand side of the web page showing the map.
#the vehicle's label will be used as the description
#the dicts below allow generating labels for the pushpins that will be mapped. Each pushpin will have a label and a description.
dict_pushpinLabel_to_desc_map = {} #this dict will associate a pushpin label to its description.
dict_vehicleId_to_pushpinLabel_map = {} #this dict will associate a vehicleId to the pushpin label it has been associated to on the map.
pushpin_url = "" #this is the url for the map with the pushpins containing the vehicle positions
total_vehicles_found = 0 #this is the total vehicles found, mapped or not
dynamodb_item_expiry_in_days = 2 #number of days after which items in dynamodb will be automatically deleted. Using DynamoDB TTL
tz_sydney = timezone('Australia/Sydney') # Set up timezone
#list all variables –end
s3 = boto3.resource('s3') # Define boto3 object to do s3 operations
dynamodb = boto3.resource('dynamodb') # Define boto3 object to do dynamodb operations
pushpinLabelsTable = dynamodb.Table(transportPositionTable)
# Initialize gtfs_realtime_pb2.FeedMessage
feed = gtfs_realtime_pb2.FeedMessage()

Remember to replace <insert your opendata api key> and <insert your bing maps api key here> with your own Open Data and Bing Maps api key. Do not include the ‘<‘ ‘>’ in the script.

Next, I will take you through the various functions that have been created to carry out specific tasks.

First up is the initialise global variables function. As you might be aware, AWS Lambdas can get reused over various invocations. During my experimentations, I found this happening and the issue I found was that my global variables were not being automatically initialised. This caused erroneous results. To circumvent this issue, I decided to write an explicit function that will initialise all global variables at the beginning of each lambda function invocation.

#most of the times, lambda functions get re-used by AWS for consequetive runs. To ensure the global variables are sanitised and do not
#cause issues because they contain values from last run, we will initialise them
def initialise_global_variables():
global lastpushpinLabel
global dict_pushpinLabel_to_desc_map
global dict_vehicleId_to_pushpinLabel_map
global pushpin_url
global total_vehicles_found
#initialise the values for the above global variables
print("Initialising global variables")
lastpushpinLabel = 0
dict_pushpinLabel_to_desc_map = {}
dict_vehicleId_to_pushpinLabel_map = {}
pushpin_url = ""
total_vehicles_found = 0

When placing each train location on the map (these will be doing using pushpins), a label will be used to identify each pushpin. Unfortunately, Bing Maps doesn’t allow more than three characters for each label. This is not enough to provide a meaningful label. The solution I devised was to use a consecutive numbering scheme for the labels and then provide a key on the website page. The key will provide a description for each label. One complication with this approach is that I need to ensure the same label is used for the same train for any lambda invocation. This is achieved by storing the label to train mapping in the Amazon DynamoDb table.

The next function downloads all label to train associations stored in Amazon DynamoDB. This ensures that labels remain consistent across lambda invocations.

#read the contents ot pushpinLabelsTable and populate pushpinLabels_LabelNum and pushpinLabels_Label dict
def load_pushpinLabels_from_dynamodb(today):
global dict_pushpinLabel_to_desc_map
global dict_vehicleId_to_pushpinLabel_map
global lastpushpinLabel
rdate = today.strftime("%Y-%m-%d")
pushpinLabelsTableContents = pushpinLabelsTable.query(
IndexName='TripDate-GSI',
KeyConditionExpression='TripDate = :tripdate',
ExpressionAttributeValues={":tripdate":rdate}
)
#check to see if there were any items returned from the above query
if len(pushpinLabelsTableContents['Items']) > 0:
#there were some items returned
#the vehicle's label could be same across different vehicles however its id is unique for the day. For this reason
#we must store the vehicle id along with the pushpinLabel and pushpinLabelDesc
for item in pushpinLabelsTableContents['Items']:
pushpinLabel = item['pushpinLabel']
pushpinLabelDesc = item['pushpinLabelDesc']
vehicleId = item['VehicleId']
inDynamodbTable = True #we will mark all entries that are being populated from dynamodb, so that when we update dynamodb
#we only write items that are missing, not everything. This saves on write capacity units
#the following fields were added progressively to dynamodb. Ensure that missing items doesn't break the code. Just add blanks if missing
try:
latitude = item['latitude']
except:
latitude = ""
try:
longitude = item['longitude']
except:
longitude = ""
try:
tripId = item['tripId']
except:
tripId = ""
dict_pushpinLabel_to_desc_map[str(pushpinLabel)] = {"desc": str(pushpinLabelDesc), "vehicleId": str(vehicleId), "inDynamodbTable": inDynamodbTable, "latitude": latitude, "longitude": longitude, "tripId": tripId}
dict_vehicleId_to_pushpinLabel_map[str(vehicleId)] = {"label": str(pushpinLabel), "desc": str(pushpinLabelDesc)}
#if a higher pushpinLabel was found in pushpinLabels table, then set the lastPushpinLabel to this
if int(pushpinLabel) > int(lastpushpinLabel):
lastpushpinLabel = int(pushpinLabel)
#else there were no results for the above query, which means there were no entries in dynamodb table for today. Let's start fresh
print("load_pushpinLabels_from_dynamodb: Loaded ", len(pushpinLabelsTableContents['Items']), "pushpinLabels from dynamodb table. lastpushpinLabel:", lastpushpinLabel)

The next function just queries Open Data API for the train positions.

def callOpenData(operator, feed):
print("callOpenData:Obtaining vehicle location for", operator,end="…")
response = requests.get(baseURL + operator, headers=headers)
feed.ParseFromString(response.content)
print("done")

The following function takes the data from the above function and processes it.

def process_feed(feed, today):
global dict_pushpinLabel_to_desc_map
global dict_vehicleId_to_pushpinLabel_map
global lastpushpinLabel
global pushpin_url
global total_vehicles_found
#we will use the python module 'time' to convert epoch time (this is what gtfsr timestamps are in) to local time
#set the timezone for time
os.environ['TZ'] = 'AEST-10AEDT-11,M10.5.0,M3.5.0'
time.tzset()
print(f'timezone set to {time.tzname}')
num_pushpin_assigned = 0
total_feed_entity = len(feed.entity)
total_vehicles_found += total_feed_entity
print('Total feed.entity:',total_feed_entity," Total Vehicles Found:",total_vehicles_found)
#As we are using Bing maps, there is a limit to the number of pushpins we can specify for our map. If we
#reach this limit, we will break out of the loop below as there is no benefit for continuing on processing
for entity in feed.entity:
if num_pushpin_assigned >= maps_bing_pushpin_limit:
break #we haev exceeded the number of pushpins that can be used with Bing Maps. Exit loop
tripupdatetimestamp_autz = time.ctime(entity.vehicle.timestamp)
vehicleId = entity.vehicle.vehicle.id
#we need to make sure that the fields used for dynamodb keys are not null. If they are then skip this record. Currently these are TripDate which is
#todays date and vehicleId. So just check vehicleId for being not null and TripDate won't be null.
if (vehicleId): #only go ahead if vehicleId is present. Otherwise just print that vehicleId for this record is missing
#lets find out if this vehicle already has a pushpinLabel assigned for today
if vehicleId in dict_vehicleId_to_pushpinLabel_map.keys():
#this vehicle already has a pushpinLabel. Get the label
pushpinLabel = dict_vehicleId_to_pushpinLabel_map[vehicleId]['label']
else:
#this vehicle doesn't have any pushpinLabel already assigned. Generate a new pushpinLabel for it
lastpushpinLabel += 1 #increment the lastpushpinLabel so that it now points to a new number
pushpinLabel = lastpushpinLabel
tripId = entity.vehicle.trip.trip_id
#the pushpinLabelDesc will be set to the vehicle's label. There have been instances where I noticed the vehicle's label is missing/null.
#In these cases, set the pushpinLabelDesc to TripId
if not entity.vehicle.vehicle.label:
pushpinLabelDesc = tripId
else:
pushpinLabelDesc = entity.vehicle.vehicle.label
#since this vehicle had not been previously assigned a pushpinLabel for today, add its details to the two dict
inDynamodbTable = False #this item has not been read from or written to dynamodb yet
latitude = entity.vehicle.position.latitude
longitude = entity.vehicle.position.longitude
dict_pushpinLabel_to_desc_map[str(pushpinLabel)] = {"desc": str(pushpinLabelDesc),"vehicleId": str(vehicleId),"inDynamodbTable":inDynamodbTable,"latitude":latitude,"longitude":longitude,"tripId":tripId}
dict_vehicleId_to_pushpinLabel_map[str(vehicleId)] = {"label": str(pushpinLabel), "desc": str(pushpinLabelDesc)}
#this vehicle will be displayed on the map. Update dict_pushpinLabel_to_desc_map for this vehicle's entry so that when the legend is
#generated, it will be coloured differently to show that it is currently displayed on the map
dict_pushpinLabel_to_desc_map[str(pushpinLabel)]['isDisplayedOnMap'] = True
#add this vehicle's details to the pushpin url
pushpin_url += "pp=" + str(entity.vehicle.position.latitude) + "," + str(entity.vehicle.position.longitude)
#based on how long ago the laststopid timestamp is, calculate if the vehicle is active or inactive and respectively assign the icon
if (datetime.datetime.now() datetime.datetime.strptime(tripupdatetimestamp_autz,'%a %b %d %H:%M:%S %Y')) > datetime.timedelta(minutes=inactive_vehicle_datetime_limit):
#this vehicle is inactive
pushpin_url += ";" + maps_inactive_vehicle_icon + ";"
else:
pushpin_url += ";" + maps_active_vehicle_icon + ";"
pushpin_url += str(pushpinLabel) + "&"
num_pushpin_assigned += 1 #increment the counter that denotes the number of pushpins added to the pushpin url
else:
print("process_feed:MissingVehicleId:VehicleWillBeSkipped:TripId:", entity.vehicle.trip.trip_id, " VehicleLabel:", entity.vehicle.vehicle.label," Latitude: ", entity.vehicle.position.latitude, " Longitude: ", entity.vehicle.position.longitude)

The function goes through each vehicle record that was returned and checks to see if the vehicle already has a label associated to it. If there is one, then this label will be used for it otherwise a new one is created. The Amazon DynamoDB table will be updated with this newly created label. This function uses the first 100 trains returned by the Open Data API, to generate a pushpin URL. This URL will be used to generate the Bing Maps showing the positions of the trains.

The following function uses the pushpin URL to create a map using Bing Maps.

def generate_vehicle_position_webpage(map_url, body, s3Bucket, s3ImageKey, imagefilename, mainHtmlfilename, today):
global dict_pushpinLabel_to_desc_map
global dict_vehicleId_to_pushpinLabel_map
global lastpushpinLabel
global total_vehicles_found
#get a handle on the s3 bucket
s3 = boto3.resource('s3')
imgObject = s3.Object(s3Bucket,s3ImageKey+imagefilename)
map_webrequest = requests.post(map_url, data=body)
#check if the maps were successfully obtained.
if map_webrequest.status_code != 200:
#there was an error. show the error
print("generate_vehicle_position_webpage:Bing Map request failed")
print("generate_vehicle_position_webpage:Reason:", map_webrequest.reason)
print("generate_vehicle_position_webpage:ErrorMessage:", map_webrequest.text)
print("generate_vehicle_position_webpage:RequestBody:", body)
#as there has been an error generating the map, display an image denoting that there has been an error
map_webrequest = requests.get(maps_error_imageURL)
else:
print("generate_vehicle_position_webpage:Bing Map request was successful. Status code:",map_webrequest.status_code)
#upload the image file to S3 bucket, set it for public read and ensure content-type is image/jpeg
print("generate_vehicle_position_webpage:Uploading map to s3 bucket")
upload_img_result = imgObject.put(Body=map_webrequest.content,ACL='public-read',ContentType='image/jpeg')
print("generate_vehicle_position_webpage:Generating landing html page:",mainHtmlfilename,end="…")
#Generate the label description using all the pushpin labels for the day. The first entry will be the header field
pushpinLabels = "<li>Label – Description <font color=" + legend_vehicle_displayed_on_map_color + ">[blue label descriptions show currently displayed vehicles]</font></li>"
for index in range(1, lastpushpinLabel + 1):
pushpinLabel_desc = dict_pushpinLabel_to_desc_map[str(index)]['desc']
#the isDisplayedOnMap attribute was added later to DynamoDB so there might be items that don't have it. This ensures that a call to get this
#attribute will not break the program
try:
isDisplayedOnMap = dict_pushpinLabel_to_desc_map[str(index)]['isDisplayedOnMap']
except:
isDisplayedOnMap = False
#for all vehicles currently displayed on map, show their label description in a different color. This makes it easy to differentiate between a vehicle
#that is currently displayed and one that was previously displayed today however it is now not mapped.
if isDisplayedOnMap:
pushpinLabels += "<font color=" + legend_vehicle_displayed_on_map_color + "><li>" + str(index) + " – " + str(pushpinLabel_desc) + "</li></font>"
else:
pushpinLabels += "<li>" + str(index) + " – " + str(pushpinLabel_desc) + "</li>"
htmlObject = s3.Object(s3Bucket,mainHtmlfilename)
htmlContent= """<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
body {
font-family: Arial;
color: white;
}
.split {
height: 100%;
width: 50%;
position: fixed;
z-index: 1;
top: 0;
overflow-x: hidden;
padding-top: 20px;
}
.left {
left: 0;
width: 70%;
background-color: #111;
}
.right {
right: 0;
width: 30%;
background-color: black;
}
.centered {
position: absolute;
top: 50%;
left: 50%;
transform: translate(-50%, -50%);
text-align: center;
}
</style>
<script>
<!–
function timedRefresh(timeoutPeriod) {
setTimeout("location.reload(true);",timeoutPeriod);
}
window.onload = timedRefresh(60000);
// –>
</script>
</head>"""
htmlContent +="""
<body>
<div class="split left">
<div class="centered">
<img src={} alt="Vehicle Location Map" width="1250" height="1000">
<p> image source {}</p>
</div>
</div>
<div class="split right">
<p>Map last updated at {} [updated every 5 min].
<br>Total vehicles found {} (at most, only the first 100 will be mapped)
<br>
<br>Icon Description
<br>Blue = active vehicles (location reported within last {} minutes)
<br>Red = inactive vehicles (location last reported at least {} minutes ago)
</p>
<ul style="list-style-type:square;">""".format(s3ImageKey+imagefilename, imagefilename, imagefilename[:4], total_vehicles_found, inactive_vehicle_datetime_limit, inactive_vehicle_datetime_limit)
htmlContent += pushpinLabels
htmlContent +="""
</ul>
</div>
</body>
</html>"""
upload_html_result = htmlObject.put(Body=htmlContent,ACL='public-read',ContentType='text/html')
print("done")
print("generate_vehicle_position_webpage:Uploaded map:",imagefilename," UploadResult:",upload_img_result)
print("generate_vehicle_position_webpage:Uploaded htmlfile:",mainHtmlfilename," UploadResult:",upload_html_result)

The function provides the pushpin URL to Bing Maps. Bing Maps returns a map showing the positions of the trains. The map is then uploaded to the Amazon S3 bucket that will serve the website. The function then goes generates a description page showing descriptions for each label. As you can imagine, each day, there are hundreds of labels being created. Not all of these labels will be displayed on the map, however they would be listed in the key area. To provide quick access to the key descriptions which show trains that are currently displayed in the map, the corresponding entries will be displayed in blue. This ensures that people don’t go on a wild goose chase, trying to locate a train on the map which might not be displayed.

The next function updates the transport position Amazon DynamoDB table with any new labels that were created during this invocation. This ensures that the labels persist for all subsequent lambda invocations.

def update_pushpinLabelsTable(tripdate):
global lastpushpinLabel
#update the pushpinLabels dynamodb table with all the pushpinLabels that were created in this invocation
print("update_pushpinLabelsTable:Uploading new pushpinLabels to dynamodb Table")
time_now = datetime.datetime.now(tz_sydney) #each item uploaded to dynamodb table will have the time it was inserted
time_now_str = time_now.strftime("%Y-%m-%d %H:%M:%S")
epoch_time_now = time_now.timestamp()
expirationTime = int(epoch_time_now + (dynamodb_item_expiry_in_days * 24 * 3600)) #convert expiry days to seconds
print("update_pushpinLabelsTable:time_now:", time_now_str," epoch_time_now:", epoch_time_now, " expirationTime:", expirationTime)
num_items_added_to_dynamodb = 0
for index in range(1, lastpushpinLabel + 1):
pushpinLabel = index
pushpinLabelDesc = dict_pushpinLabel_to_desc_map[str(index)]['desc']
vehicleId = dict_pushpinLabel_to_desc_map[str(index)]['vehicleId']
inDynamodbTable = dict_pushpinLabel_to_desc_map[str(index)]['inDynamodbTable']
latitude = dict_pushpinLabel_to_desc_map[str(index)]['latitude']
longitude = dict_pushpinLabel_to_desc_map[str(index)]['longitude']
tripId = dict_pushpinLabel_to_desc_map[str(index)]['tripId']
#only write back to dynamodb table those entries that are new. Dynamodb items to be regarded as immutable and should not be changed.
#using TTL (which is set to attribute ExpirationTime) allows for easy cleanup of items as we don't want items longer than 24 hours as they
#are not being mapped (default expirationTime has been set to 2 days)
if (not inDynamodbTable):
try:
pushpinLabelsTable.put_item(
Item={
'TripDate': str(tripdate),
'VehicleId': str(vehicleId),
'tripId': str(tripId),
'pushpinLabel': str(pushpinLabel),
'pushpinLabelDesc': str(pushpinLabelDesc),
'latitude': str(latitude),
'longitude': str(longitude),
'TimeAdded': time_now_str,
'ExpirationTime': expirationTime
}
)
num_items_added_to_dynamodb += 1
except Exception as e:
print("update_pushpinLabelsTable:Error with put_item operation ",str(e))
print("update_pushpinLabelsTable:TripDate:",str(tripdate)," VehicleId:",str(vehicleId)," pushpinLabel:",str(pushpinLabel)," pushpinLabelDesc:",str(pushpinLabelDesc)," Latitude:",str(latitude)," Longitude:",str(longitude)," TimeAdded:",time_now_str," ExpirationTime:",expirationTime)
num_items_added_to_dynamodb += 1
print("update_pushpinLabelsTable:Uploaded ",num_items_added_to_dynamodb," new pushpinLabel(s) to dynamodb table. LastpushpinLabel:",lastpushpinLabel)

Now that we know what all the functions do, lets move on to the handler function that the tps lambda will call. The handler function will coordinate all the functions.

def run(event, context):
print(datetime.datetime.now(), 'Started')
today = datetime.datetime.now(tz_sydney)
initialise_global_variables()
load_pushpinLabels_from_dynamodb(today)
#loop through each operator and get their vehicle positions
for operator in operators:
callOpenData(operator, feed)
process_feed(feed, today)
rdate = today.strftime("%Y-%m-%d")
time_now = datetime.datetime.now(tz_sydney)
mapImageName = time_now.strftime("%Y-%m-%dT%H%M") + '.jpg'
#get fhe image file and upload it to s3
generate_vehicle_position_webpage(maps_bing_prefix + maps_bing_apikey,pushpin_url[:1], bucketName, bucketImagesKey, mapImageName, HtmlLandingpagename, rdate)
update_pushpinLabelsTable(rdate)

The handler function calls the respective functions to get the following tasks done (in the order listed below)

  • Initialises the global variables
  • downloads the previously associated labels from the Amazon DynamoDB table
  • calls Open Data API to get the latest position of the trains.
  • the train data is then processed, and the pushpin URL generated
  • Using the pushpin URL, the map is generated using Bing Maps.
  • A landing page is created. This webpage shows the map along with a key to show descriptions for each label.
  • Finally, all labels that were created within this lambda invocation are uploaded to the Amazon DynamoDB table. This ensures that for all subsequent invocations of the lambda function, the respective trains get the same label assigned to them.

The full tps_vehiclepos.py file can be downloaded from https://gist.github.com/nivleshc/03cb06bd6ae9cb969192af0ee2a1a15b

That’s it! Now you have a good idea about how the AWS resources were generated and how the data was acquired, processed and then visualised.

Cost to run the solution

When I started developing this solution, to view what type of data was being provided by Open Data API, I tried to ingest everything into DynamoDB. This was a VERY VERY bad idea as it cost me quite a lot. However, since then I have modified my code to only ingest and store fields that are required, into Amazon DynamoDB table. This has drastically reduced by running costs. Currently I am being charged approximately USD0.05 or less per day. You can run this easily within your free tier without incurring any additional costs (just as a precaution, monitor your costs to ensure there are no surprises).

As mentioned at the beginning of this blog, the final result can be seen at https://sls-tps-website-dev.s3-ap-southeast-2.amazonaws.com/vehiclelocation.html. The webpage is refreshed every one minute however the map is generated every five minutes. I will keep this project running for at least three months. So, if you want to check it out, you have ample time.

I have extremely enjoyed creating this project, I hope you all enjoy it as well. Till the next time, Enjoy!

Display Raspberry Pi Metrics using AWS CloudWatch

Background

I am all for cloud computing, however there are some things, in my view, that still need an on-premises presence. One such is devices that allow you to securely connect to your home network. For this, I use a Raspberry Pi running OpenVPN server. OpenVPN is an awesome tool and apart from securely connecting to my home network, it also allows me to securely tunnel my network traffic via my home network when I am connected to an unsecured network.

However, over the last few days, I have been having issues with my vpn connections. It would intermittently disconnect, and at times I will have to try a few times before the connection was re-established. At first it was a nuisance, however lately it has become a bigger issue. Finally, I decided to fix the issue!

After spending some time on it, guess what the problem turned out to be? A few weeks back I had installed a software on my Raspberry Pi for testing purposes. I forgot to uninstall it and now for some reason it was hogging the CPU! As I didn’t need this software, the quick fix was to simply uninstall it.

This got me thinking. There must be a better way to monitor the CPU/Memory/Disk space usage on my Raspberry PI instead of logging onto it every now and then, or worse, when things broke. I could install monitoring tools on it which could notify me when certain thresholds were breached. However, this meant adding more workloads to my Raspberry Pi, something which I wasn’t too keen on doing. I finally decided to publish the metrics to AWS CloudWatch and create some alarms in it.

In this blog, I will list the steps that I followed, to publish my Raspberry Pi metrics to AWS CloudWatch. Without taking much more time, let’s get started.

Create an AWS user for the AWS CloudWatch agent

The AWS CloudWatch agent that will run on our Raspberry Pi needs to be able to authenticate with our AWS account, before it can upload any metrics.

To enable this, create an IAM user with programmatic access and assign the CloudWatchAgentServerPolicy directly to it. When you are creating this IAM user, keep a note of the secret access key that is displayed at the end of the user creation process. If you lose these keys, there is no way to recover them, the only option will be to regenerate it.

For detailed instructions on creating this IAM user, visit https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html

Downloading and installing the AWS CloudWatch agent

With the IAM user done, let’s proceed to installing the AWS CloudWatch agent on the Raspberry Pi. The available agents can be downloaded from https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html

Knowing that my Raspberry Pi runs Raspbian as the operating system, which is a variant of Debian, I proceeded to download the ARM64 version of the .deb file. This is when the fun started!

Running the command

dpkg -I -E ./amazon-cloudwatch-agent.deb

gave me the following error

package architecture (arm64) does not match system (armhf)

Interesting, according to the error, it seemed that my Raspberry Pi has an armhf (arm hard float) architecture, which is not supported by the standard AWS CloudWatch agents. For those interested in the various Debian ports, this webpage lists all of them https://www.debian.org/ports/#portlist-released

This latest discovery put my planning into a tailspin!

After spending some time searching, I came across https://github.com/awslabs/collectd-cloudwatch . This described a plugin for collectd, which would allow me to publish the Raspberry Pi metrics to AWS CloudWatch!

Horary! I was back on track again! Below is a record of what I did next.

  1. On my Raspberry Pi, I installed collectd using
    sudo apt-get install collectd
  2. I then downloaded the installation script for the AWS CloudWatch plugin using the following command
    wget https://github.com/awslabs/collectd-cloudwatch/blob/master/src/setup.py
  3. Once downloaded, I used chmod to make the script executable using the following command
    chmod u+x setup.py
  4. If you look through the script, you will notice that it tries to detect the linux distribution for the system it is running on, and then uses the respective installer commands to install the plugin. Digging abit further, I found that the way it detects the linux distribution is by looking through the files matching the pattern /etc/*-release.

    When I looked at all files fitting the name pattern /etc/*release, the only file I found was /etc/os-release which was a symbolic link to /usr/lib/os-release

    Opening the file /usr/lib/os-release, I noticed that the name of the distribution that was installed on my Raspberry Pi was “Raspbian GNU/Linux

    Comparing this to the script setup.py, I found that it wasn’t one of the supported distributions. Fear not because this is easily remedied!

    So here is what you do.

    Open setup.py in your favorite editor and scroll down to where the following section is

    DISTRIBUTION_TO_INSTALLER = {
      "Ubuntu": APT_INSTALL_COMMAND,
      "Red Hat Enterprise Linux Server": YUM_INSTALL_COMMAND,
      "Amazon Linux AMI": YUM_INSTALL_COMMAND,
      "Amazon Linux": YUM_INSTALL_COMMAND,
      "CentOS Linux": YUM_INSTALL_COMMAND,
    }

    Add the line

    "Raspbian GNU": APT_INSTALL_COMMAND,

    after

    "CentOS Linux": YUM_INSTALL_COMMAND

    You should now have the following

    DISTRIBUTION_TO_INSTALLER = {
      "Ubuntu": APT_INSTALL_COMMAND,
      "Red Hat Enterprise Linux Server": YUM_INSTALL_COMMAND,
      "Amazon Linux AMI": YUM_INSTALL_COMMAND,
      "Amazon Linux": YUM_INSTALL_COMMAND,
      "CentOS Linux": YUM_INSTALL_COMMAND,
      "Raspbian GNU": APT_INSTALL_COMMAND,
    }
  5. Run setup.py. The script seems to be customised to run within an Amazon EC2 instance because it tries to gather information by querying the instance metadata urls. Ignore these errors and enter the information that is requested. Below are the questions that will be asked

    When asked for your region, enter your AWS region. For me, this is ap-southeast-2
    When asked, enter the hostname of the Raspberry Pi
    Next, you will be asked about the AWS credentials to connect to AWS CloudWatch. Enter the credentials for the user that was created above
    Unless you are using a proxy server, answer none to "Enter a proxy server name" and "Enter a proxy server port"
    At the next prompt for "Include the Auto-Scaling Group name as a metric dimension" choose No
    For "Include the Fixed Dimension as a metric dimension" prompt choose No
    At the next prompt for "Enable high resolution" choose No
    For "Enable flush internal", leave this at "Default 60s"
    The last question asks how to install the CloudWatch plugin. Choose "Add plugin to existing configuration"
  6. Now that all the questions have been answered, you must select the metrics that have to be published to AWS CloudWatch. To check which metrics can be published, open the file

    /opt/collectd-plugins/cloudwatch/config/blocked-metrics.

    From the above file, select the metrics that you want to be published to AWS CloudWatch, and copy them into the file

     /opt/collectd-plugins/cloudwatch/config/whitelist.conf
  7. After the whitelist has been populated, restart the collectd agent so that it can read the updated settings. To do this, issue the following commandt
    sudo service collectd restart
  8. Thats it! Give it approximately five minutes and the Raspberry Pi metrics should be populated inside AWS CloudWatch. To check, login to AWS CloudWatch and under Metrics, you should see a custom namespace for collectd. This is the metrics that were sent from your Raspberry Pi.

Here is a screenshot of the CPU metrics that my Raspberry Pi uploaded to AWS CloudWatch

AWS CloudWatch Raspberry Pi Metrics

 

If you want to be alerted when a certain metric reaches a particular threshold, you can create an alarm within AWS CloudWatch, that notifies you when it happens.

Thats it folks! Till the next time, Enjoy!