Display Control Plane API Operations using Amazon CloudWatch Logs Insights

Introduction

For small organisations that cannot afford to spend much on their network security, moving to the cloud enables them to easily uplift their security posture. The organisation can concentrate on innovating and scaling their workloads, while AWS provides them with a secure environment to use. More information regarding AWS security and compliance can be found at https://docs.aws.amazon.com/whitepapers/latest/aws-overview/security-and-compliance.html

The sentence above is not entirely accurate. The security of workloads in AWS is a shared responsibility between AWS and the customer. AWS is responsible for “Security of the cloud”. This means that AWS is responsible for protecting the infrastructure that runs the workloads (hardware, software, networking, physical facilities). The customer is responsible for “Security in the cloud”. This means that the customer, based on the service they select, must perform all the necessary security configuration and management tasks to keep the workload secure. A good place to learn more about this is at https://aws.amazon.com/compliance/shared-responsibility-model/

As a good security practice, one must always monitor all the activities that happen in their cloud environment, especially those that involve the management of resources. For instance, if one notices a lot of large Amazon EC2 instances being provisioned, this could possibly be an indication of a breach (or someone authorised is provisioning these without notifying others). The management operations performed on resources in your AWS account are referred to as Control Plane operations.

There are many commercial products that can help you with monitoring your AWS environment. However, you can quite adequately benefit from the tools that are natively provided by AWS as well.

In this blog, I will take you through the steps of using Amazon CloudWatch Logs Insights, to easily display the Control Plane operations, in a meaningful way.

Implementation

Let’s start:

  1. Open the AWS Console and then navigate to the AWS CloudTrail service page. Change to the appropriate AWS region.
  2. Create a new Trail to record just Management events. Ensure it is applied to all the regions and that it delivers events to Amazon CloudWatch Logs as well. Below is a screenshot of the required settings

  3. Open the Amazon CloudWatch service page and ensure it is in the same region as the AWS CloudTrail that was just configured (in step 2 above)
  4. Click on Dashboards and then click on Create dashboard. Give the dashboard a meaningful name (I called my dashboard ControlPlaneOperations-Dashboard)
  5. In the next screen, from the top menu, click on Add widget. Another screen will open. Select Query results and then click on Configure

  6. In the next screen, use the drop-down arrow (pointed by the red arrow below) to select the CloudWatch Log group that was configured for the new CloudTrail in step 1 above.

    In the formula section (denoted by the red rectangle), delete everything and replace it with the text below (the screenshot above already has the correct formula)

    fields eventTime, userIdentity.userName, userIdentity.accessKeyId, sourceIPAddress, awsRegion, eventSource, eventName | sort eventTime desc
    

    The above formula directs Amazon CloudWatch Log Insights to display the event time, user name and access key of the identity that performed the control plane operation, the ip address from where the operation was performed, the AWS region inside which the operation was performed, the event’s source and name. The results are sorted based on the event time in descending order.

    Next, set the time range for events that CloudWatch Logs Insights must process. To configure this, pick the appropriate duration from those displayed on the top right (as displayed inside the green rectangle in the screenshot above). For my setup, I chose 1hr.

    Once completed, click Create widget

  7. The next screen should look similar the screenshot below. Click Save dashboard

     

    Amazon CloudWatch Logs Insights - Create Dashboard

  8. Your Amazon CloudWatch dashboard is now complete. To refresh the events, you can press the refresh button (pointed by the red arrow in the screenshot
  9. Amazon CloudWatch Logs Insights - Refresh Events
  10. You can also enable auto refresh of the events by clicking the small arrow beside the refresh button. You will get a menu option similar to the screenshot below.

    Tick Auto refresh and choose the Refresh interval of your choice.

  11. Pro Tip 1 – if you want your dashboard to be displayed under the Favorite section when you open the Amazon CloudWatch service page, go into the Dashboards section of Amazon CloudWatch and click on Favorite (star) beside your dashboard name.
  12. Pro Tip 2 – If you want your dashboard to appear on the default Amazon CloudWatch service page, rename your dashboard to CloudWatch-Default.
  13. Pro Tip 3 = at the beginning of each row In your dashboard, you will notice a small arrow head. If you click on the arrow head, it expands that event and provides additional information.

Thats it! You should now have a dashboard similar to the one below that shows the control plane operations as they happen.

Amazon CloudWatch Logs Insights Dashboard

At times, I found there to be approximately five minutes of delay between when the event happened and when it was displayed. This could be due to the delay between when the event was generated and when that service that generated it delivered the logs to AWS CloudTrail.

The dashboard should allow you to easily monitor any suspicious control plane activities in your AWS account.

I hope the above was useful. Till the next time, Enjoy!

Creating a Contact Center in minutes using Amazon Connect

Background

In my previous blog (https://nivleshc.wordpress.com/2019/10/09/managing-amazon-ec2-instances-using-amazon-ses/), I showed how we can manage Amazon EC2 instances using emails. However, what if you wanted to go further than that? What if, instead of sending an email, you instead wanted to dial in and check the status of or start/stop your Amazon EC2 instances?

In this blog, I will show how I used the above as a foundation to create my own Contact Center. I enriched the experience by including an additional option for the caller, to be transferred to a human agent. All this in minutes! Still skeptical? Follow on and I will show you how I did all of this using Amazon Connect.

High Level Solution Design

Below is the high-level solution design for the Contact Center I built.

The steps (as denoted by the numbers in the diagram above) are explained below

  1. The caller dials the Direct Inward Dial (DID) number associated with the Amazon Connect instance
  2. Amazon Connect answers the call
  3. Amazon Connect invokes the AWS Lambda function to authenticate the caller.
  4. The AWS Lambda function authenticates the caller by checking their callerID against the entries stored in the authorisedCallers DynamoDB table. If there is a match, the first name and last name stored against the callerID is returned to Amazon Connect. Otherwise, an “unauthorised user” message is returned to Amazon Connect.
  5. If the caller is unauthorised, Amazon Connect informs them of this and hangs up the call.
  6. If the caller is authorised, Amazon Connect uses the first name and last name provided by AWS Lambda function to personalise a welcome message for them. Amazon Connect then provides the caller with two options:
      •  (6a) press 1 to get the status of the Amazon EC2 instances. If this is pressed, Amazon Connect invokes an AWS Lambda function to get the status of the Amazon EC2 instances and plays the results to the caller
      • (6b) press 2 to talk to an agent. If this is pressed, Amazon Connect places the call in a queue,  where it will be answered by the next available agent

     

Preparation

My solution requires the following components

  • Amazon DynamoDB table to store authorised callers (an item in this table will have the format phonenumber, firstname,  lastname)
  • AWS Lambda function to authenticate callers
  • AWS Lambda function to get the status of all Amazon EC2 instances in the region

I created the following AWS CloudFormation template to provision the above resources.

The above AWS CloudFormation template can be downloaded from https://gist.github.com/nivleshc/926259dbbab22dd4890e0708cf488983

Implementation

Currently AWS CloudFormation does not support Amazon Connect. The implementation must be done manually.

Leveraging on my own experience with setting up Amazon Connect solutions,  I observed that there are approximately three stages that are required to get a Contact Center up and running. These are:

  • Provisioning an Amazon Connect instance – this is straight forward and essentially is where an Amazon Connect instance is provisioned and made ready for your use
  • Configuring the Amazon Connect instance – this contains all the tasks to customise the Amazon Connect instance. It includes the configuration of the Direct Inward Dial (DID), hours or operations for the Contact Center, Routing profiles, users etc
  • Creating a custom Contact flow – a Contact flow defines the customer experience of your Contact Center, from start to finish. Amazon Connect provides a few default Contact flows however it is highly recommended that you create one that aligns with your own business requirements.

Follow along and I will show you how to go about setting up each of the above mentioned stages.

Provision the Amazon Connect Instance

  1. From the AWS Console, open the Amazon Connect service. Select the Sydney region (or a region of your choice, however do keep in mind that at the moment, Amazon Connect is only available in a few regions)
  2. Enter an Access URL for your Amazon Connect Instance. This URL will be used to access the Amazon Connect instance once it has been provisioned.
  3. In the next screen, create an administrator account for this Amazon Connect instance
  4. The next prompt is for Telephony options. For my solution, I selected the following:
    1. Incoming calls: I want to handle incoming calls with Amazon Connect
    2. Outgoing calls: I want to make outbound calls with Amazon Connect
  5. In the next screen, Data Storage options are displayed. For my solution, I left everything as default.
  6. In the next screen, review the configuration and then click Create instance

Configure the Amazon Connect Instance

After the Amazon Connect instance has been successfully provisioned, use the following steps to configure it:

  1. Claim a phone number for your Amazon Connect Instance. This is the number that users will be calling to interact with your Amazon Connect instance (for claiming non toll free local numbers, you must open a support case with AWS, to prove that you have a local business in the country where you are trying to claim the phone number. Claiming a local toll-free number is easier however it is more expensive)
  2. Create some Hour of operation profiles. These will be used when creating a queue
  3. Create a queue. Each queue can have different hours of operation assigned
  4. Create a routing profile. A user is associated with a routing profile, which defines their inbound and outbound queues.
  5. Create users. Once created, assign the users to a predefined security profile (administrator, agent etc) and also assign them to a specific routing profile

Create a custom Contact flow

A Contact flow defines the customer experience of your Contact Center, from start to finish. By default, Amazon Connect provides a few Contact flows that you can use. However, it is highly recommended that you create one that suits your own business requirements.

To create a new Contact flow, follow these steps:

  • Login to your Amazon Connect instance using the Access URL and administrator account (you can also access your Amazon Connect instance using the AWS Console and then click on Login as administrator)
  • Once logged in, from the left-hand side menu, go to Routing and then click on Contact flows
  • In the next screen, click on Create contact flow
  • Use the visual editor to create your Contact flow

Once the Contact flow has been created, attach it to your Direct Inward Dial (DID) phone number by using the following steps:

  • from the left-hand side menu, click on Routing and then Phone numbers.
  • Click on the respective phone number and change its Contact flow / IVR to the Contact flow you want to attach to this phone number.

Below is a screenshot of the Contact flow I created for my solution. It shows the flow logic I used and you can easily replicate it for your own environment. The red rectangles show where the AWS Lambda functions (mentioned in the pre-requisites above) are used.

This is pretty much all that is required to get your Contact Center up and running. It took me approximately thirty minutes from start to finish (this does not include the time required to provision the Amazon DynamoDB tables and AWS Lambda functions). However, I would recommend spending time on your Contact flows as this is brains of the operation. This must be done in conjunction with someone who understands the business really well and knows the outcomes that must be achieved by the Contact Center solution. There is a lot that can be done here and the more time you invest in your Contact flow, the better outcomes you will get.

The above is just a small part of what Amazon Connect is capable of. For its full set of capabilities, refer to https://aws.amazon.com/connect/

So, if you have been dreaming of building your own Contact Center, however were worried about the cost or effort required? Wait no more! You can now easily create one in minutes using Amazon Connect and pay for only what you use and tear it down if you don’t need it anymore. However, before you start, I would strongly recommend that you get yourself familiar with the Amazon Connect pricing model. For example – you get charged a daily rate for any claimed phone numbers that are attached to your Amazon Connect Instance (this is similar to a phone line rental charge). Full pricing is available at https://aws.amazon.com/connect/pricing/).

I hope the above has given you some insights into Amazon Connect. Till the next time, Enjoy!

Managing Amazon EC2 Instances using Amazon SES

Background

Most people know Amazon Simple Email Service (SES) just as a service for sending out emails. However, did you know that you can use it to receive emails as well? If this interests you, more information is available at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email.html.

In this blog I will show how I provisioned a solution to manage my Amazon EC2 instances using emails. The solution uses Amazon SES and AWS Lambda. Now, some of you might be saying, can’t you just use the AWS console or app for this? Well, yes you can, however for me personally, logging into an AWS console just to get the status of my Amazon EC2 instances, or to start/stop them was more effort than I deemed necessary. The app surely makes this task trivial, however the main purpose of this blog is to showcase the capabilities of Amazon SES

Solution Architecture

Below is the high-level design for my solution.

The individual steps (labelled using numbers) are described below

  1. The admin sends an email to an address attached to an Amazon SES rule
  2. Amazon SES receives the email, performs a spam and virus check. If the email passes the check, Amazon SES invokes the manageInstances AWS Lambda function, passing the contents of the email to it (unfortunately the contents of the body are not passed)
  3. The manageInstances AWS Lambda function authenticates the sender based on the from address (this is a very rudimentary authentication system. A stronger authentication mechanism must be used if this solution is to be deployed in a production environment – maybe include a multi-factor authentication system). It extracts the command from the email’s subject and executes it
  4. The manageInstances AWS Lambda function uses Amazon SES to send the response of the command back to the admin
  5. Amazon SES delivers the email containing the command’s output to the admin

Prerequisites

To use Amazon SES for receiving incoming emails, first verify your domain within it and then point your domain’s DNS MX entry to your region’s Amazon SES endpoint. Full Instructions to carry this out can be found at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-getting-started.html

Implementation

Lambda Function

The Lambda function is created first as it is required for the Amazon SES Rules.

The Lambda function carries out the following tasks

  • authenticates the sender by comparing the email’s from address with a predefined list of approved senders. To prevent a situation where the Lambda function can be inadvertently used as a spam bot, all emails from senders not on the approved list will be dropped.
  • checks if the specified command is in the list of commands that is currently supported. If yes, then the command is executed, and the output sent back to the admin. If the command is unsupported, a reply stating that an invalid command was specified is sent back to the admin

Here are the attributes of the Lambda function I created

Function name: manageInstances
Runtime:             Python 3.6
Region:                North Virginia (us-east-1) This is to ensure that the Lambda function and the Amazon SES rules are in the same region
Role:                     The role that is used by the Lambda function must have the following permissions attached to it

             ec2:DescribeInstances
             ec2:StartInstances
             ec2:StopInstances
             SES:SendEmail

Here is the code for the AWS Lambda function (set the approvedSenders list to contain the email address of approved senders)

The AWS Lambda function code can be downloaded from https://gist.github.com/nivleshc/f9b32a14d9e662701c3abcbb8f264306

Amazon SES Email Receiving Rule

Next, the Amazon SES Email Receiving rule that handles the incoming emails must be created. Please note that currently Amazon SES is supported in a few regions only. For my solution, I used the North Virginia (us-east-1) region.

Below are the steps to create the rule

  • Open Simple Email Service service page from within the AWS console (ensure you are in the correct AWS region)
  • From the left-hand side menu, navigate down to the Email Receiving section and then click Rule Sets.
  • The right-hand side of the screen will show the currently defined rule sets. I used the predefined default-rule-set.
  • Click on View Active Rule Set and then in the next screen click Create Rule.
  • In the next screen, for the recipient address, enter the email address to which emails will be sent to, to carry out the commands (the email domain has to correspond to the domain that was verified with Amazon SES, as part of the prerequisites mentioned above)
  • In the next screen, for actions, select Lambda.
  • Select the name of the Lambda function that was created to manage the instances from the drop down (for me, this was manageInstances)
  • Ensure the Invocation type is set to Event.
  • You do not need to set the SNS topic, however if you need to know when this Amazon SES action is carried out, select the appropriate SNS topic (you will need to create an SNS topic and subscribe to it using your email address)
  • Click Next Step.
  • In the next screen, provide a name for the rule. Ensure the options Enabled and Enable spam and virus scanning are ticked.
  • Click Next Step and then review the settings.
  • Click Create Rule.

Usage

The solution, once implemented supports the following commands

help   - provides information about the commands and their syntax
status - provides the status of all Amazon EC2 instances in the region that the Lambda function is running in. It lists the instance-id and name of those instances (name is derived from the tag with the key Name)
start {instance-id} - starts the Amazon EC2 instance that has the specified instance-id
stop {instance-id}  - stops the Amazon EC2 instance that has the specified instance-id

To use, send an email from an approved sender’s email to the email address attached to Amazon SES.

The table below shows, what the subject must be, for each command.

Command Subject
Help help
Status for instances in us-east-1 status
Start an instance with instance-id i-0e7e011b42e814465 start i-0e7e011b42e814465
Stop an instance with instance-id i-0e7e011b42e814465 stop i-0e7e011b42e814465

The output of the status command is in the following format

<instance-id> <instance-name> <status> <private-ip> <public-ip>

for example
i-03a1ab124f554z805 LinuxServer01 Running 172.16.31.10 52.10.100.34

The only problem with the solution is that all commands are performed on Amazon EC2 instances running in the same AWS region as the Lambda function. What if you wanted to carry out the commands on another region?

For the keen eyed, you would have spotted the Easter egg I hid in the Lambda function code. Here is what the subject must be if the command is to be carried out in an AWS region different to where the Lambda function is running (simply provide the AWS region at the end of the command)

Command Subject
Help help
Status for instances in ap-southeast-2 (Sydney) status ap-southeast-2
Start an instance with instance-id i-0e7e011b42e814465 in ap-southeast-2 (Sydney) region start i-0e7e011b42e814465 ap-southeast-2
Stop an instance with instance-id i-0e7e011b42e814465 in ap-southeast-2 (Sydney) region stop i-0e7e011b42e814465 ap-southeast-2

There you go! Now you can keep an eye on and control your Amazon EC2 instances with just your email.

A good use case can be when you are commuting and need to RDP into your Windows Amazon EC2 instance from your mobile (I am guilty of doing this at times). You can quickly start the Amazon EC2 instance, get its public ip address, and then connect using RDP.  Once finished, you can shut down the instance to ensure you don’t get charged after that.

I hope this blog was useful to you. Till the next time, Enjoy!

Using Serverless Framework and AWS to map Near-Realtime Positions of Trains

Background

A couple of months back, I found out about the Open Data initiative from Transport for New South Wales. This is an awesome undertaking, to provide data to developers and other interested parties, so that they can develop great applications. For those interested, the Open Data hub can be accessed at https://opendata.transport.nsw.gov.au.

I had been playing with data for a few months now and when I looked through the various APIs that I could access via the Open Data hub, I became extremely interested.

In this blog, I will take you through one of my mini projects based off the data at Open Data Hub. I will be using the Public Transport – Vehicle Positions API to plot the near realtime positions of Sydney trains on a map. The API provides access to more than just train position data, however to keep things simple, I will concentrate on only trains in this blog.

Solution Architecture

As I am a huge fan of serverless, I decided to architect my solution with as much serverless components as possible. The diagram below shows a high-level architecture of how the Transport Positioning System (this is what I will call my solution, TPS for short) will be created.

Let’s go through the steps (as marked in the diagram above)

  1. The lambda function will query the Open Data API every 5 minutes for the position of all trains
  2. After the data has been received, the lambda function will go through each record and assign a label to each train. To ensure the labels are consistent across each lambda invocation, the train to label association will be stored in an Amazon DynamoDB table. The lambda function will query the table to check if a train has already been allocated a label. If it has, then that label will be used. Otherwise, a new label will be created, and the Amazon DynamoDB table will be updated to store this new train to label association.
  3. I found Bing Maps to be much easier (and cheaper) to use for plotting items on a map. The only disadvantage is that it can, at most show 100 points (called pushpins) on the map. The lambda function will go through the first 100 items returned from Open Data API and using the labels that were found/created in step 2, create a pushpin url that will be used to generate a map of the location of the first 100 trains. This url will then be used to generate the map by sending a request to Bing Maps.
  4. The lambda function will then create a static webpage that displays the map showing the position of the trains, along with a description that provides more information about the labels used for each train (for example, label 1 could have a description of “19:10 Central Station to Penrith Station”). The label description is set to the train’s label, which is obtained from the query results of the Open Data API query.

This is what I cooked up earlier

Be warned! This blog is quite lengthy as a lot was done to make this solution work. However, if you would rather see the final result before delving into the details, check it out https://sls-tps-website-dev.s3-ap-southeast-2.amazonaws.com/vehiclelocation.html.

Screenshot of TPS

Above is a screenshot of TPS in action. It is a static webpage that is being generated every 5 minutes, showing the position of trains. I will keep the lambda function running for at least three months, so you have plenty of time available to check it out.

Okay, let’s get our hands dirty and start coding.

Prerequisites

Before we start, the following must be in place

  1. You must have an AWS account. If you don’t have one already, you can sign up for a free tier at https://aws.amazon.com/free/
  2. You must have Serverless Framework installed on your computer. If you don’t have, it follow the details at https://serverless.com/framework/docs/getting-started/
  3. Setup the AWS access keys and secret access key that Serverless Framework will use to provision resources into your AWS account. Instructions to get this done can be obtained from https://serverless.com/framework/docs/providers/aws/guide/credentials/
  4. After items 1 – 3 have been completed, create a python runtime Serverless Framework service (my service is called tps)
  5. Within the tps service folder, install the following Serverless Framework plugins
    1. serverless-python-requirements – this plugin adds all the required python modules into a zip file containing our lambda function script, which then gets uploaded to AWS (the required python modules must be defined in the requirement.txt file)
    2. serverless-prune-plugin – this plugin ensures that only the specified number of lambda function versions exist.
  6. Serverless-python-requirements requires a file called requirements.txt. For this solution, create a file called requirements.txt at the root of the service folder and put the following lines inside it
    requests==2.22.0
    gtfs-realtime-bindings==0.0.6
  7. Register an account with Transport for New South Wales Open Data Hub (https://opendata.transport.nsw.gov.au/). This is free. Once registered, login to the Open Data Hub portal and then under My Account, click on Applications and then create an Application that has permissions to Public Transport – Realtime Vehicle Positions API. Note down the API key that is generated as it will be used by the python script later.
  8. Create an account with Bing Maps, use the Website licence plan. This will provide 125,000 billable transactions of generating maps per calendar year at no charge. For generating a map every 5 minutes, this is more than enough. Note down the API key that is provided. Details on creating a Bing Maps account is available at https://www.microsoft.com/en-us/maps/licensing/options

This project has two parts to it. The first is to create the AWS resources that will host our project. For this, we will be using Serverless Framework to create our AWS Lambda function, Amazon Simple Storage Service (S3) bucket, Amazon DynamoDB table, AWS CloudWatch Logs and AWS CloudWatch Events.

The second part is the API query and data ingestion from the Open Data API. The next sections will cover each of these parts.

AWS Resource Creation

As previously mentioned, we will use Serverless Framework to create our AWS resources. Serverless Framework uses serverless.yml to specify which resources need to be created. This file is created by default whenever a Serverless Framework service is created.

In this section, I will take you through the serverless.yml file I used for this project.

The file starts like this.

It defines the service name, plugins and variables that will be used in this file (notice the plugins serverless-python-requirements and serverless-prune-plugin)

The following default values have been configured in the above serverless.yml file.

  • application name is set to tps
  • environment is set to dev
  • Amazon CloudWatch logs is set to 14 days retention
  • the AWS region is set to ap-southeast-2 (Sydney)
  • the Amazon DynamoDB table is set to transportPosition
  • the billing mode for the Amazon DynamoDB table will be set to PAY PER REQUEST.

The next section defines the details for the cloud provider where resources will be provisioned.

As previously mentioned, I am using AWS. The lambda function will use python 3.7 runtime. The deployment bucket to host all artefacts is also defined. This is an Amazon Simple Storage Service (S3) bucket. Ensure that this S3 bucket exists before deploying the serverless service.

The next section defines the IAM role that will be created for the lambda function.

The specified IAM role will allow the lambda function to carry out all required operations on Amazon Dynamodb table (currently the IAM role permits all DynamoDB actions, however if required, this can be tightened) and to upload objects to the Amazon S3 bucket that will host the static website.

The next section provides instructions on what to include and exclude when creating the serverless package.

Now we come to the important sections for within serverless.yml. The next section defines the lambda function, its handler and what events will trigger it.

For the tps lambda function, the handler function is at src.tps_vehiclepos.run (this means that there is a subfolder within the service folder called src, within which there is a python file called tps_vehiclepos. Inside this file is a function called run). The lambda function will be run every 5 minutes. To achieve this, I am using AWS Events. Two environment variables are also being passed to the lambda function (BUCKET and TRANSPORTPOSITION_TABLE)

The last section defines all the resources that will be created by the Serverless Framework.

The following resources will be created

  • Amazon S3 bucket. This bucket will store the Bing Maps that show the train positions. It will also serve the website which will be used to display the position maps.
  • Amazon DynamoDB table. The table will be used to store details for trains found in the Open Data API query. Note that we are also using the DynamoDB TTL feature. Since we don’t need to retain items older than a day, this will allow us to easily prune the Amazon DynamoDB table, to reduce costs.

The full serverless.yml file can be downloaded from https://gist.github.com/nivleshc/951ff2f235a89d58d4abec6de91ef738

Generating the map

In this section we will go through the script that does all the magic, which is querying the Open Data API, processing the data, generating the map and then displaying the results in a webpage.

The script is called tps_vehiclelocation.py. I will discuss the script in parts below.

The first section lists the python modules that will be imported. It also defines the variables that will be used within the script.

Remember to replace <insert your opendata api key> and <insert your bing maps api key here> with your own Open Data and Bing Maps api key. Do not include the ‘<‘ ‘>’ in the script.

Next, I will take you through the various functions that have been created to carry out specific tasks.

First up is the initialise global variables function. As you might be aware, AWS Lambdas can get reused over various invocations. During my experimentations, I found this happening and the issue I found was that my global variables were not being automatically initialised. This caused erroneous results. To circumvent this issue, I decided to write an explicit function that will initialise all global variables at the beginning of each lambda function invocation.

When placing each train location on the map (these will be doing using pushpins), a label will be used to identify each pushpin. Unfortunately, Bing Maps doesn’t allow more than three characters for each label. This is not enough to provide a meaningful label. The solution I devised was to use a consecutive numbering scheme for the labels and then provide a key on the website page. The key will provide a description for each label. One complication with this approach is that I need to ensure the same label is used for the same train for any lambda invocation. This is achieved by storing the label to train mapping in the Amazon DynamoDb table.

The next function downloads all label to train associations stored in Amazon DynamoDB. This ensures that labels remain consistent across lambda invocations.

The next function just queries Open Data API for the train positions.

The following function takes the data from the above function and processes it.

The function goes through each vehicle record that was returned and checks to see if the vehicle already has a label associated to it. If there is one, then this label will be used for it otherwise a new one is created. The Amazon DynamoDB table will be updated with this newly created label. This function uses the first 100 trains returned by the Open Data API, to generate a pushpin URL. This URL will be used to generate the Bing Maps showing the positions of the trains.

The following function uses the pushpin URL to create a map using Bing Maps.

The function provides the pushpin URL to Bing Maps. Bing Maps returns a map showing the positions of the trains. The map is then uploaded to the Amazon S3 bucket that will serve the website. The function then goes generates a description page showing descriptions for each label. As you can imagine, each day, there are hundreds of labels being created. Not all of these labels will be displayed on the map, however they would be listed in the key area. To provide quick access to the key descriptions which show trains that are currently displayed in the map, the corresponding entries will be displayed in blue. This ensures that people don’t go on a wild goose chase, trying to locate a train on the map which might not be displayed.

The next function updates the transport position Amazon DynamoDB table with any new labels that were created during this invocation. This ensures that the labels persist for all subsequent lambda invocations.

Now that we know what all the functions do, lets move on to the handler function that the tps lambda will call. The handler function will coordinate all the functions.

The handler function calls the respective functions to get the following tasks done (in the order listed below)

  • Initialises the global variables
  • downloads the previously associated labels from the Amazon DynamoDB table
  • calls Open Data API to get the latest position of the trains.
  • the train data is then processed, and the pushpin URL generated
  • Using the pushpin URL, the map is generated using Bing Maps.
  • A landing page is created. This webpage shows the map along with a key to show descriptions for each label.
  • Finally, all labels that were created within this lambda invocation are uploaded to the Amazon DynamoDB table. This ensures that for all subsequent invocations of the lambda function, the respective trains get the same label assigned to them.

The full tps_vehiclepos.py file can be downloaded from https://gist.github.com/nivleshc/03cb06bd6ae9cb969192af0ee2a1a15b

That’s it! Now you have a good idea about how the AWS resources were generated and how the data was acquired, processed and then visualised.

Cost to run the solution

When I started developing this solution, to view what type of data was being provided by Open Data API, I tried to ingest everything into DynamoDB. This was a VERY VERY bad idea as it cost me quite a lot. However, since then I have modified my code to only ingest and store fields that are required, into Amazon DynamoDB table. This has drastically reduced by running costs. Currently I am being charged approximately USD0.05 or less per day. You can run this easily within your free tier without incurring any additional costs (just as a precaution, monitor your costs to ensure there are no surprises).

As mentioned at the beginning of this blog, the final result can be seen at https://sls-tps-website-dev.s3-ap-southeast-2.amazonaws.com/vehiclelocation.html. The webpage is refreshed every one minute however the map is generated every five minutes. I will keep this project running for at least three months. So, if you want to check it out, you have ample time.

I have extremely enjoyed creating this project, I hope you all enjoy it as well. Till the next time, Enjoy!

Display Raspberry Pi Metrics using AWS CloudWatch

Background

I am all for cloud computing, however there are some things, in my view, that still need an on-premises presence. One such is devices that allow you to securely connect to your home network. For this, I use a Raspberry Pi running OpenVPN server. OpenVPN is an awesome tool and apart from securely connecting to my home network, it also allows me to securely tunnel my network traffic via my home network when I am connected to an unsecured network.

However, over the last few days, I have been having issues with my vpn connections. It would intermittently disconnect, and at times I will have to try a few times before the connection was re-established. At first it was a nuisance, however lately it has become a bigger issue. Finally, I decided to fix the issue!

After spending some time on it, guess what the problem turned out to be? A few weeks back I had installed a software on my Raspberry Pi for testing purposes. I forgot to uninstall it and now for some reason it was hogging the CPU! As I didn’t need this software, the quick fix was to simply uninstall it.

This got me thinking. There must be a better way to monitor the CPU/Memory/Disk space usage on my Raspberry PI instead of logging onto it every now and then, or worse, when things broke. I could install monitoring tools on it which could notify me when certain thresholds were breached. However, this meant adding more workloads to my Raspberry Pi, something which I wasn’t too keen on doing. I finally decided to publish the metrics to AWS CloudWatch and create some alarms in it.

In this blog, I will list the steps that I followed, to publish my Raspberry Pi metrics to AWS CloudWatch. Without taking much more time, let’s get started.

Create an AWS user for the AWS CloudWatch agent

The AWS CloudWatch agent that will run on our Raspberry Pi needs to be able to authenticate with our AWS account, before it can upload any metrics.

To enable this, create an IAM user with programmatic access and assign the CloudWatchAgentServerPolicy directly to it. When you are creating this IAM user, keep a note of the secret access key that is displayed at the end of the user creation process. If you lose these keys, there is no way to recover them, the only option will be to regenerate it.

For detailed instructions on creating this IAM user, visit https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html

Downloading and installing the AWS CloudWatch agent

With the IAM user done, let’s proceed to installing the AWS CloudWatch agent on the Raspberry Pi. The available agents can be downloaded from https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html

Knowing that my Raspberry Pi runs Raspbian as the operating system, which is a variant of Debian, I proceeded to download the ARM64 version of the .deb file. This is when the fun started!

Running the command

dpkg -I -E ./amazon-cloudwatch-agent.deb

gave me the following error

package architecture (arm64) does not match system (armhf)

Interesting, according to the error, it seemed that my Raspberry Pi has an armhf (arm hard float) architecture, which is not supported by the standard AWS CloudWatch agents. For those interested in the various Debian ports, this webpage lists all of them https://www.debian.org/ports/#portlist-released

This latest discovery put my planning into a tailspin!

After spending some time searching, I came across https://github.com/awslabs/collectd-cloudwatch . This described a plugin for collectd, which would allow me to publish the Raspberry Pi metrics to AWS CloudWatch!

Horary! I was back on track again! Below is a record of what I did next.

  1. On my Raspberry Pi, I installed collectd using
    sudo apt-get install collectd
  2. I then downloaded the installation script for the AWS CloudWatch plugin using the following command
    wget https://github.com/awslabs/collectd-cloudwatch/blob/master/src/setup.py
  3. Once downloaded, I used chmod to make the script executable using the following command
    chmod u+x setup.py
  4. If you look through the script, you will notice that it tries to detect the linux distribution for the system it is running on, and then uses the respective installer commands to install the plugin. Digging abit further, I found that the way it detects the linux distribution is by looking through the files matching the pattern /etc/*-release.

    When I looked at all files fitting the name pattern /etc/*release, the only file I found was /etc/os-release which was a symbolic link to /usr/lib/os-release

    Opening the file /usr/lib/os-release, I noticed that the name of the distribution that was installed on my Raspberry Pi was “Raspbian GNU/Linux

    Comparing this to the script setup.py, I found that it wasn’t one of the supported distributions. Fear not because this is easily remedied!

    So here is what you do.

    Open setup.py in your favorite editor and scroll down to where the following section is

    DISTRIBUTION_TO_INSTALLER = {
      "Ubuntu": APT_INSTALL_COMMAND,
      "Red Hat Enterprise Linux Server": YUM_INSTALL_COMMAND,
      "Amazon Linux AMI": YUM_INSTALL_COMMAND,
      "Amazon Linux": YUM_INSTALL_COMMAND,
      "CentOS Linux": YUM_INSTALL_COMMAND,
    }

    Add the line

    "Raspbian GNU": APT_INSTALL_COMMAND,

    after

    "CentOS Linux": YUM_INSTALL_COMMAND

    You should now have the following

    DISTRIBUTION_TO_INSTALLER = {
      "Ubuntu": APT_INSTALL_COMMAND,
      "Red Hat Enterprise Linux Server": YUM_INSTALL_COMMAND,
      "Amazon Linux AMI": YUM_INSTALL_COMMAND,
      "Amazon Linux": YUM_INSTALL_COMMAND,
      "CentOS Linux": YUM_INSTALL_COMMAND,
      "Raspbian GNU": APT_INSTALL_COMMAND,
    }
  5. Run setup.py. The script seems to be customised to run within an Amazon EC2 instance because it tries to gather information by querying the instance metadata urls. Ignore these errors and enter the information that is requested. Below are the questions that will be asked

    When asked for your region, enter your AWS region. For me, this is ap-southeast-2
    When asked, enter the hostname of the Raspberry Pi
    Next, you will be asked about the AWS credentials to connect to AWS CloudWatch. Enter the credentials for the user that was created above
    Unless you are using a proxy server, answer none to "Enter a proxy server name" and "Enter a proxy server port"
    At the next prompt for "Include the Auto-Scaling Group name as a metric dimension" choose No
    For "Include the Fixed Dimension as a metric dimension" prompt choose No
    At the next prompt for "Enable high resolution" choose No
    For "Enable flush internal", leave this at "Default 60s"
    The last question asks how to install the CloudWatch plugin. Choose "Add plugin to existing configuration"
  6. Now that all the questions have been answered, you must select the metrics that have to be published to AWS CloudWatch. To check which metrics can be published, open the file

    /opt/collectd-plugins/cloudwatch/config/blocked-metrics.

    From the above file, select the metrics that you want to be published to AWS CloudWatch, and copy them into the file

     /opt/collectd-plugins/cloudwatch/config/whitelist.conf
  7. After the whitelist has been populated, restart the collectd agent so that it can read the updated settings. To do this, issue the following commandt
    sudo service collectd restart
  8. Thats it! Give it approximately five minutes and the Raspberry Pi metrics should be populated inside AWS CloudWatch. To check, login to AWS CloudWatch and under Metrics, you should see a custom namespace for collectd. This is the metrics that were sent from your Raspberry Pi.

Here is a screenshot of the CPU metrics that my Raspberry Pi uploaded to AWS CloudWatch

AWS CloudWatch Raspberry Pi Metrics

 

If you want to be alerted when a certain metric reaches a particular threshold, you can create an alarm within AWS CloudWatch, that notifies you when it happens.

Thats it folks! Till the next time, Enjoy!

Using Amazon Alexa to drive a radio-controlled car – Part 1

Introduction

Over the Easter holidays, while watching my son play with his radio-controlled toy car, I had a strange thought pop into my head. Instead of using the sticks on the remote control, won’t it be cool to control the car by using just your voice? You could tell the car to move forward, backward, left or right. What if you could save all the moves you have asked the car to take so far and then at a later time, get the car to replay all those moves?

Now, that would be a car I would love to play with!

In this blog, I will introduce the high-level design for accomplishing the above-mentioned goal. Then over the next few blogs I will take you through the steps to transform the high-level design into a working prototype.

Hardware Requirements

For this prototype, I settled on using the following hardware devices

  • Amazon Echo Dot – this will be used to process my voice commands
  • Raspberry Pi 3 with a GPIO expansion Breadboard
  • A set of four 5v Relay Board Module
  • A radio-controlled race car
  • A soldering iron, solder wire and a digital multimeter

Design considerations

To make the prototype work, I decided to create an Amazon Alexa Skill called race car. This will be used to process my voice commands.

Challenge #1: How would I control the radio-controlled car?

I found two options for this

1. Completely bypass the remote control and send the radio frequency instructions directly to the race car

2. Emulate the button presses on the remote control so that it “thinks” someone is pressing those buttons and then it sends the appropriate radio frequency instructions to the race car

Option Chosen: I chose option 2 because it required the least amount of work. For this option, the only thing I needed to figure out was what happened when a button was pressed. After some experimentation, I found the contacts on the printed circuit board (PCB) of the remote control which I could open and close the contacts on, to emulate the button presses.

Challenge #2: I will use a python script running on a Raspberry Pi 3 within my home network to emulate the button presses on the remote control. How will I get the Amazon Alexa Skill to connect to my Raspberry Pi 3 which is running on my internal home network?

Solution: I found a neat trick at https://developer.amazon.com/blogs/post/Tx14R0IYYGH3SKT/flask-ask-a-new-python-framework-for-rapid-alexa-skills-kit-development . To expose my internal Raspberry Pi 3 python script to the Amazon Alexa Skill, I will use ngrok (https://ngrok.com) to create a secure tunnel between my Raspberry Pi 3 and the ngrok service. This provides me with an HTTPS endpoint within ngrok’s domain, which forwards any requests directed at the ngrok endpoint to the python script running on my internal Raspberry Pi 3 using the secure tunnel.

High Level Design for the prototype

Using the above-mentioned design considerations, the below schematic was developed to create the prototype.

Let’s go through each of the steps (denoted by the numbers) to better understand the design.

1. The user will invoke the race car Amazon Alexa Skill and ask to either move the car in a certain direction, save all the movements that have been requested so far, or run a previously saved set of movements.

2. The Alexa device (Amazon Echo Dot) will record the audio from the user and send it to the Alexa Cloud for processing. Alexa Cloud converts the audio into JSON using Natural Language Processing (NLP). Based on the invocation name, it will pass the JSON file to the race car Amazon Alexa Skill.

3. The race car Amazon Alexa Skill will check to ensure that the intent supplied by the user is valid. Once confirmed, the race car Amazon Alexa Skill will pass the JSON to the endpoint defined for the skill. In our case, this is an endpoint that is hosted at ngrok (https://ngrok.com)

4. The ngrok endpoint will receive the JSON file from the race car Amazon Alexa Skill and then forward it using the secure tunnel to the python script running on the Raspberry Pi 3 within the home network. The python script will use the Flask-Ask framework to process the intents from the Alexa Skills Kit (more information for Flask-Ask can be obtained from https://flask-ask.readthedocs.io/en/latest/)

5. If the user requested to save all the car movements carried out so far, then the python script will write the movements to a table within Amazon DynamoDB.

6. If the user requested to load a previously saved set of movements, then the python script will read the movements from the table within Amazon DynamoDB.

7. If the use requested to either load a previously saved set of movements or to move the car in a certain direction, the python script will emulate the appropriate button presses on the remote control.

8. The remote control will translate the emulated button presses into radio frequency instructions and send them to the car. The car will receive these instructions and move accordingly.

To give you a sneak peek of the prototype, checkout the video at https://youtu.be/4SMYDhuri0Q (there are some minor bugs with the car movement which I intend on getting fixed as soon as possible).

In the next blog in this series, we will go through the process of “hacking” the remote control and also setting up the Raspberry Pi 3 ancillary hardware.

I hope to see you then.

Till then, enjoy!

Using Ansible to create an inventory of your AWS resources

Background

I was recently at a customer site, to perform an environment review of their AWS real-estate. As part of this engagement, I was going to do an inventory of all their AWS resources. Superficially, this sounds like an easy task, however when you consider the various regions that resources can be provisioned into, the amount of work required for a simple inventory can easily escalate.

Not being a big fan of manual work, I started to look at ways to automate this task. I quickly settled on Ansible as the tool of choice and not long after, I had two ansible playbooks ready (the main and the worker playbook) to perform the inventory.

In this blog, I will introduce the two ansible playbooks that I wrote. The first playbook is the main actor. This is where the variables are defined. This playbook iterates over the specified AWS regions, calling the worker playbook each time, to check if any resources have been provisioned in these regions. The output is written to comma separated value (csv) files (I am using semi-colons instead of commas), which can be easily imported into Microsoft Excel (or any spreadsheet program of your choice) for analysis.

Introducing the Ansible playbooks

The playbooks have been configured to check the following AWS resources

  • Virtual Private Cloud (vpc)
  • Subnets within the VPCs
  • Internet Gateways
  • Route Tables
  • Security Groups
  • Network Access Control Lists
  • Customer Gateways
  • Virtual Private Gateways
  • Elastic IP Addresses
  • Elastic Compute Cloud Instances
  • Amazon Machine Images that were created
  • Elastic Block Store Volumes
  • Elastic Block Store Snapshots
  • Classic Load Balancers
  • Application Load Balancers
  • Relational Database Service Instances
  • Relational Database Service Snapshots
  • Simple Storage Service (S3) Buckets

The table below provides details for the two ansible playbooks.

Filename Purpose
ansible-aws-inventory-main.yml This is the controller playbook. It iterates over each of the specified regions, calling the worker playbook to check for any resources that are provisioned in these regions.
ansible-aws-inventory-worker.yml This playbook does all the heavy lifting. It checks for any provisioned resources in the region that is provided to it by the controller playbook

Let’s go through each of the sections in the main ansible playbook (ansible-aws-inventory-main.yml), to get a better understanding of what it does.

First off, the variables that will be used are defined

aws_regions – this defines all the AWS regions which will be checked for provisioned resources

verbose – to display the results both on screen and to write it to file, set this to true. Setting this to false just writes the results to file.

owner_id – this is the account id for the AWS account that is being inventoried. It is used to retrieve all the Amazon Machine Images (AMI) that are owned by this account

Next, the column headers for each of the csv files is defined.

After this, the output filenames are defined. Do note that the filenames use timestamps (for when the playbook is run) as prefixes. This ensures that they don’t overwrite any output files from previous runs.

When I was generating the inventory list, at times I found that I needed only a subset of resource types inventoried, instead of all (for instance when I was looking for only EC2 instances). For this reason, I found it beneficial to have boolean variables to either enable or disable inventory checks for specific resource types.

The next section lists boolean variables that control if a particular resource type should be checked or not. Set this to true if it is to be checked and false if it is to be skipped. You can set this to your own preference.

After all the variables have been defined, the tasks that will be carried out are configured.

The first task initialises the output csv files with the column headers.

Once the initialisation has been completed, the inventory process is started by looping through each of the specified AWS regions and calling the worker ansible playbook to check for provisioned resources.

The last task displays the path for the output files.

The main ansible playbook (ansible-aws-inventory-main.yml) can be downloaded from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de

The worker playbook (ansible-aws-inventory-worker.yml) has the following format

  • go through each of the defined resource types and confirm that it is to be checked (checks for a particular resource type are enabled using the boolean variable that is defined in the main playbook)
  • If checks are enabled for that particular resource type, find all provisioned resources of that type in the region provided by the main ansible playbook
  • write the results to the respective output file
  • if verbose is enabled, write the results to screen

The worker file (ansible-aws-inventory-worker.yml) can be downloaded from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500

Running the ansible playbooks

Use the following steps to run the above mentioned ansible playbooks to perform an inventory of your AWS account.

1. On a computer that has ansible installed, create a folder and name it appropriately (for instance inventory)

2. Download ansible-aws-inventory-main.yml from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de and put it in the folder that was created in step 1 above

3. Download ansible-aws-inventory-worker.yml from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500 and put it in the folder that was created in step 1 above

4. Download the ansible inventory file from https://gist.github.com/nivleshc/bc2e300fe1d2779ecc15c0876fc4db62 , rename it to hosts and put it in the folder that was created in step 1 above

5. Customise ansible-aws-inventory-main.yml by adding your account id as the owner_id and change the output folder by updating the output_root_folder variable. If you need to disable inventory for certain resource types, you can set the respective boolean variable to false.

6. Create a user account with access keys enabled within your AWS account. For checking all the resources defined in the playbook, at a minimum, the account must have the following permissions assigned

AmazonVPCReadOnlyAccess
AmazonEC2ReadOnlyAccess
ElasticLoadBalancingReadOnly
AmazonRDSReadOnlyAccess
AmazonS3ReadOnlyAccess

7. Open a command line and then run the following to configure environment variables with credentials of the account that was created in step 6 above (the following commands are specific to MacOS)

export AWS_ACCESS_KEY_ID="xxxxx"
export AWS_SECRET_ACCESS_KEY="xxxxxxx"

8. There is a possibility that you might encounter an error with boto complaining that it is unable to access region us-west-3. To fix this, define the following environment variable as well

export BOTO_USE_ENDPOINT_HEURISTICS=True

9. Run the ansible playbook using the following command line

ansible-playbook -i hosts ansible-aws-inventory-main.yml

Depending on how many resources are being inventoried, the playbook can take anywhere from five to ten minutes to complete. So, sit back and relax, while the playbook runs.

I found a bug with the ansible “aws s3 bucket facts” module. It ignores the region parameter and instead of returning s3 buckets in a specific region, it returns buckets in all regions. Due to this, the s3 buckets csv file will have the same buckets repeated in all the regions.

Hope you enjoy the above ansible playbooks and they make your life much easier when trying to find all resources that are deployed within your AWS account.

Till the next time, enjoy!

Feature Photo by Samuel Zeller on Unsplash