Display Raspberry Pi Metrics using AWS CloudWatch

Background

I am all for cloud computing, however there are some things, in my view, that still need an on-premises presence. One such is devices that allow you to securely connect to your home network. For this, I use a Raspberry Pi running OpenVPN server. OpenVPN is an awesome tool and apart from securely connecting to my home network, it also allows me to securely tunnel my network traffic via my home network when I am connected to an unsecured network.

However, over the last few days, I have been having issues with my vpn connections. It would intermittently disconnect, and at times I will have to try a few times before the connection was re-established. At first it was a nuisance, however lately it has become a bigger issue. Finally, I decided to fix the issue!

After spending some time on it, guess what the problem turned out to be? A few weeks back I had installed a software on my Raspberry Pi for testing purposes. I forgot to uninstall it and now for some reason it was hogging the CPU! As I didn’t need this software, the quick fix was to simply uninstall it.

This got me thinking. There must be a better way to monitor the CPU/Memory/Disk space usage on my Raspberry PI instead of logging onto it every now and then, or worse, when things broke. I could install monitoring tools on it which could notify me when certain thresholds were breached. However, this meant adding more workloads to my Raspberry Pi, something which I wasn’t too keen on doing. I finally decided to publish the metrics to AWS CloudWatch and create some alarms in it.

In this blog, I will list the steps that I followed, to publish my Raspberry Pi metrics to AWS CloudWatch. Without taking much more time, let’s get started.

Create an AWS user for the AWS CloudWatch agent

The AWS CloudWatch agent that will run on our Raspberry Pi needs to be able to authenticate with our AWS account, before it can upload any metrics.

To enable this, create an IAM user with programmatic access and assign the CloudWatchAgentServerPolicy directly to it. When you are creating this IAM user, keep a note of the secret access key that is displayed at the end of the user creation process. If you lose these keys, there is no way to recover them, the only option will be to regenerate it.

For detailed instructions on creating this IAM user, visit https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html

Downloading and installing the AWS CloudWatch agent

With the IAM user done, let’s proceed to installing the AWS CloudWatch agent on the Raspberry Pi. The available agents can be downloaded from https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html

Knowing that my Raspberry Pi runs Raspbian as the operating system, which is a variant of Debian, I proceeded to download the ARM64 version of the .deb file. This is when the fun started!

Running the command

dpkg -I -E ./amazon-cloudwatch-agent.deb

gave me the following error

package architecture (arm64) does not match system (armhf)

Interesting, according to the error, it seemed that my Raspberry Pi has an armhf (arm hard float) architecture, which is not supported by the standard AWS CloudWatch agents. For those interested in the various Debian ports, this webpage lists all of them https://www.debian.org/ports/#portlist-released

This latest discovery put my planning into a tailspin!

After spending some time searching, I came across https://github.com/awslabs/collectd-cloudwatch . This described a plugin for collectd, which would allow me to publish the Raspberry Pi metrics to AWS CloudWatch!

Horary! I was back on track again! Below is a record of what I did next.

  1. On my Raspberry Pi, I installed collectd using
    sudo apt-get install collectd
  2. I then downloaded the installation script for the AWS CloudWatch plugin using the following command
    wget https://github.com/awslabs/collectd-cloudwatch/blob/master/src/setup.py
  3. Once downloaded, I used chmod to make the script executable using the following command
    chmod u+x setup.py
  4. If you look through the script, you will notice that it tries to detect the linux distribution for the system it is running on, and then uses the respective installer commands to install the plugin. Digging abit further, I found that the way it detects the linux distribution is by looking through the files matching the pattern /etc/*-release.

    When I looked at all files fitting the name pattern /etc/*release, the only file I found was /etc/os-release which was a symbolic link to /usr/lib/os-release

    Opening the file /usr/lib/os-release, I noticed that the name of the distribution that was installed on my Raspberry Pi was “Raspbian GNU/Linux

    Comparing this to the script setup.py, I found that it wasn’t one of the supported distributions. Fear not because this is easily remedied!

    So here is what you do.

    Open setup.py in your favorite editor and scroll down to where the following section is

    DISTRIBUTION_TO_INSTALLER = {
      "Ubuntu": APT_INSTALL_COMMAND,
      "Red Hat Enterprise Linux Server": YUM_INSTALL_COMMAND,
      "Amazon Linux AMI": YUM_INSTALL_COMMAND,
      "Amazon Linux": YUM_INSTALL_COMMAND,
      "CentOS Linux": YUM_INSTALL_COMMAND,
    }

    Add the line

    "Raspbian GNU": APT_INSTALL_COMMAND,

    after

    "CentOS Linux": YUM_INSTALL_COMMAND

    You should now have the following

    DISTRIBUTION_TO_INSTALLER = {
      "Ubuntu": APT_INSTALL_COMMAND,
      "Red Hat Enterprise Linux Server": YUM_INSTALL_COMMAND,
      "Amazon Linux AMI": YUM_INSTALL_COMMAND,
      "Amazon Linux": YUM_INSTALL_COMMAND,
      "CentOS Linux": YUM_INSTALL_COMMAND,
      "Raspbian GNU": APT_INSTALL_COMMAND,
    }
  5. Run setup.py. The script seems to be customised to run within an Amazon EC2 instance because it tries to gather information by querying the instance metadata urls. Ignore these errors and enter the information that is requested. Below are the questions that will be asked

    When asked for your region, enter your AWS region. For me, this is ap-southeast-2
    When asked, enter the hostname of the Raspberry Pi
    Next, you will be asked about the AWS credentials to connect to AWS CloudWatch. Enter the credentials for the user that was created above
    Unless you are using a proxy server, answer none to "Enter a proxy server name" and "Enter a proxy server port"
    At the next prompt for "Include the Auto-Scaling Group name as a metric dimension" choose No
    For "Include the Fixed Dimension as a metric dimension" prompt choose No
    At the next prompt for "Enable high resolution" choose No
    For "Enable flush internal", leave this at "Default 60s"
    The last question asks how to install the CloudWatch plugin. Choose "Add plugin to existing configuration"
  6. Now that all the questions have been answered, you must select the metrics that have to be published to AWS CloudWatch. To check which metrics can be published, open the file

    /opt/collectd-plugins/cloudwatch/config/blocked-metrics.

    From the above file, select the metrics that you want to be published to AWS CloudWatch, and copy them into the file

     /opt/collectd-plugins/cloudwatch/config/whitelist.conf
  7. After the whitelist has been populated, restart the collectd agent so that it can read the updated settings. To do this, issue the following commandt
    sudo service collectd restart
  8. Thats it! Give it approximately five minutes and the Raspberry Pi metrics should be populated inside AWS CloudWatch. To check, login to AWS CloudWatch and under Metrics, you should see a custom namespace for collectd. This is the metrics that were sent from your Raspberry Pi.

Here is a screenshot of the CPU metrics that my Raspberry Pi uploaded to AWS CloudWatch

AWS CloudWatch Raspberry Pi Metrics

 

If you want to be alerted when a certain metric reaches a particular threshold, you can create an alarm within AWS CloudWatch, that notifies you when it happens.

Thats it folks! Till the next time, Enjoy!

Advertisements

Using Ansible to create an inventory of your AWS resources

Background

I was recently at a customer site, to perform an environment review of their AWS real-estate. As part of this engagement, I was going to do an inventory of all their AWS resources. Superficially, this sounds like an easy task, however when you consider the various regions that resources can be provisioned into, the amount of work required for a simple inventory can easily escalate.

Not being a big fan of manual work, I started to look at ways to automate this task. I quickly settled on Ansible as the tool of choice and not long after, I had two ansible playbooks ready (the main and the worker playbook) to perform the inventory.

In this blog, I will introduce the two ansible playbooks that I wrote. The first playbook is the main actor. This is where the variables are defined. This playbook iterates over the specified AWS regions, calling the worker playbook each time, to check if any resources have been provisioned in these regions. The output is written to comma separated value (csv) files (I am using semi-colons instead of commas), which can be easily imported into Microsoft Excel (or any spreadsheet program of your choice) for analysis.

Introducing the Ansible playbooks

The playbooks have been configured to check the following AWS resources

  • Virtual Private Cloud (vpc)
  • Subnets within the VPCs
  • Internet Gateways
  • Route Tables
  • Security Groups
  • Network Access Control Lists
  • Customer Gateways
  • Virtual Private Gateways
  • Elastic IP Addresses
  • Elastic Compute Cloud Instances
  • Amazon Machine Images that were created
  • Elastic Block Store Volumes
  • Elastic Block Store Snapshots
  • Classic Load Balancers
  • Application Load Balancers
  • Relational Database Service Instances
  • Relational Database Service Snapshots
  • Simple Storage Service (S3) Buckets

The table below provides details for the two ansible playbooks.

Filename Purpose
ansible-aws-inventory-main.yml This is the controller playbook. It iterates over each of the specified regions, calling the worker playbook to check for any resources that are provisioned in these regions.
ansible-aws-inventory-worker.yml This playbook does all the heavy lifting. It checks for any provisioned resources in the region that is provided to it by the controller playbook

Let’s go through each of the sections in the main ansible playbook (ansible-aws-inventory-main.yml), to get a better understanding of what it does.

First off, the variables that will be used are defined

aws_regions – this defines all the AWS regions which will be checked for provisioned resources

verbose – to display the results both on screen and to write it to file, set this to true. Setting this to false just writes the results to file.

owner_id – this is the account id for the AWS account that is being inventoried. It is used to retrieve all the Amazon Machine Images (AMI) that are owned by this account

Next, the column headers for each of the csv files is defined.

After this, the output filenames are defined. Do note that the filenames use timestamps (for when the playbook is run) as prefixes. This ensures that they don’t overwrite any output files from previous runs.

When I was generating the inventory list, at times I found that I needed only a subset of resource types inventoried, instead of all (for instance when I was looking for only EC2 instances). For this reason, I found it beneficial to have boolean variables to either enable or disable inventory checks for specific resource types.

The next section lists boolean variables that control if a particular resource type should be checked or not. Set this to true if it is to be checked and false if it is to be skipped. You can set this to your own preference.

After all the variables have been defined, the tasks that will be carried out are configured.

The first task initialises the output csv files with the column headers.

Once the initialisation has been completed, the inventory process is started by looping through each of the specified AWS regions and calling the worker ansible playbook to check for provisioned resources.

The last task displays the path for the output files.

The main ansible playbook (ansible-aws-inventory-main.yml) can be downloaded from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de

The worker playbook (ansible-aws-inventory-worker.yml) has the following format

  • go through each of the defined resource types and confirm that it is to be checked (checks for a particular resource type are enabled using the boolean variable that is defined in the main playbook)
  • If checks are enabled for that particular resource type, find all provisioned resources of that type in the region provided by the main ansible playbook
  • write the results to the respective output file
  • if verbose is enabled, write the results to screen

The worker file (ansible-aws-inventory-worker.yml) can be downloaded from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500

Running the ansible playbooks

Use the following steps to run the above mentioned ansible playbooks to perform an inventory of your AWS account.

1. On a computer that has ansible installed, create a folder and name it appropriately (for instance inventory)

2. Download ansible-aws-inventory-main.yml from https://gist.github.com/nivleshc/64ea7201fb0ba8cb6f87d06adc6152de and put it in the folder that was created in step 1 above

3. Download ansible-aws-inventory-worker.yml from https://gist.github.com/nivleshc/bedd2c440c816ebc86dbaeddef50d500 and put it in the folder that was created in step 1 above

4. Download the ansible inventory file from https://gist.github.com/nivleshc/bc2e300fe1d2779ecc15c0876fc4db62 , rename it to hosts and put it in the folder that was created in step 1 above

5. Customise ansible-aws-inventory-main.yml by adding your account id as the owner_id and change the output folder by updating the output_root_folder variable. If you need to disable inventory for certain resource types, you can set the respective boolean variable to false.

6. Create a user account with access keys enabled within your AWS account. For checking all the resources defined in the playbook, at a minimum, the account must have the following permissions assigned

AmazonVPCReadOnlyAccess
AmazonEC2ReadOnlyAccess
ElasticLoadBalancingReadOnly
AmazonRDSReadOnlyAccess
AmazonS3ReadOnlyAccess

7. Open a command line and then run the following to configure environment variables with credentials of the account that was created in step 6 above (the following commands are specific to MacOS)

export AWS_ACCESS_KEY_ID="xxxxx"
export AWS_SECRET_ACCESS_KEY="xxxxxxx"

8. There is a possibility that you might encounter an error with boto complaining that it is unable to access region us-west-3. To fix this, define the following environment variable as well

export BOTO_USE_ENDPOINT_HEURISTICS=True

9. Run the ansible playbook using the following command line

ansible-playbook -i hosts ansible-aws-inventory-main.yml

Depending on how many resources are being inventoried, the playbook can take anywhere from five to ten minutes to complete. So, sit back and relax, while the playbook runs.

I found a bug with the ansible “aws s3 bucket facts” module. It ignores the region parameter and instead of returning s3 buckets in a specific region, it returns buckets in all regions. Due to this, the s3 buckets csv file will have the same buckets repeated in all the regions.

Hope you enjoy the above ansible playbooks and they make your life much easier when trying to find all resources that are deployed within your AWS account.

Till the next time, enjoy!

Feature Photo by Samuel Zeller on Unsplash

Using Ansible to deploy an AWS environment

Background

Over the past few weeks, I have been looking at various automation tools for AWS. One tool that seems to get a lot of limelight is Ansible, an open source automation tool from Red Hat. I decided to give it a go, and to my amazement, I was surprised at how easy it was to learn Ansible, and how powerful it can be.

All that one must do is to write up a list of tasks using YAML notation in a file (called a playbook) and get Ansible to execute it. Ansible reads the playbook and executes the tasks in the order that they are written. Here is the biggest advantage, there are no agents to be installed on the managed computers! Ansible connects to each of the managed computers using ssh or winrm.

Another nice feature of Ansible is that it supports third party modules. This allows Ansible to be extended to support many of the services that it natively does not understand.

In this blog, we will be focusing on one of the third-party modules, the AWS module. Using this, we will use Ansible to deploy an environment within AWS.

Scenario

For this blog, we will use Ansible to provision an AWS Virtual Private Cloud (VPC) in the North Virginia (us-east-1) region. Within this VPC, we will create a public and a private subnet. We will then deploy a jumphost in the public subnet and a server within the private subnet.

Below is a diagram depicting what will be done.

Figure 1: Environment that will be deployed within AWS using Ansible Playbook

Preparation

The computer that is used to run Ansible to manage all other computers is referred to as the control machine. Currently, Ansible can be run from any machine with Python 2 (version 2.7) or Python 3 (version 3.5 or higher) installed. The Ansible control machine can run the following operating systems

  • Red Hat
  • Debian
  • CentOS
  • macOS
  • any of the BSD variants

Note: Currently windows operating system is not supported for running the control machine.

For this blog, I am using a MacBook to act as the control machine.

Before we run Ansible, we need to get a few things done. Let’s go through them now.

  1. We will use pip (Python package manager) to install Ansible. If you do not already have pip installed, run the following command to install it
    sudo easy_install pip
  2. With pip installed, use the following command to install Ansible
    sudo pip install ansible

    For those that are not using macOS for their control machine, you can get the relevant installation commands from https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html.

  3. Next, we must install the AWS Command Line Interface (CLI) tools. Use the following command for this.
    sudo pip install awscli

    More information about the AWS CLI tools is available at https://aws.amazon.com/cli/

  4. To provision items within AWS, we need to provide Ansible with a user account that has the necessary permissions. Using the AWS console, create a user account ensuring it is assigned an access key and a secret access key. At a minimum, this account must have the following policies assigned to it.
    AmazonEC2FullAccess
    AmazonVPCFullAccess

    Note: As this is a privileged user account, please ensure that the access key and secret access key is kept in a safe place.

  5. To provision AWS Elastic Compute Cloud (EC2) instances, we require key pairs created in the region that the EC2 instances will be deployed in. Ensure that you already have key pairs for the North Virginia (us-east-1) region. If not, please create them.

Instructions

Create an Ansible Playbook

Use the following steps to create an Ansible playbook to provision an AWS environment.

Open your favourite YAML editor and paste the following code

The above code instructs Ansible that it should connect to the local computer, to run all the defined tasks. This means that Ansible modules will use the local computer to connect to AWS APIs in order to carry out the tasks.

Another thing to note is that we are declaring two variables. These will be used later in the playbook.

  • vpc_region – this is the AWS region where the AWS environment will be provisioned (currently set to us-east-1)
  • my_useast1_key – provide the name of your key pair for the us-east-1 region that will be used to provision EC2 instances

Next, we will define the tasks that Ansible must carry out. The format of the tasks is as follows

  • name – this gives a descriptive name for the task
  • module name – this is the module that Ansible will use to carry out the task
  • module Parameters – these are parameters passed to the module, to carry out the specific task
  • register – this is an optional keyword and is used to record the output that is returned from the module, after the task has been carried out.

Copy the following lines of code into your YAMl file.

The above code contains two tasks.

  • the first task creates an AWS Virtual Private Cloud (VPC) using the ec2_vpc_net module. The output of this module is recorded in the variable ansibleVPC using the register command
  • the second task outputs the contents of the variable ansibleVPC using the debug command (this displays the output of the previous task)

Side Note

  • Name of the VPC has been set to ansibleVPC
  • The CIDR block for the VPC has been set to 172.32.0.0/16
  • The state keyword controls what must be done to the VPC. In our case, we want it created and to exist, as such, the value for state has been set to present.
  • The region is being set by referencing the variable that was defined earlier. Variables are referenced with the notation “{{ variable name }}”

Copy the following code to create an AWS internet gateway and associate it with the newly created VPC. The second task in the below code displays the result of the internet gateway creation.

The next step is to create the public and private subnets. However, instead of hardcoding the availability zones into which these subnets will be deployed, we will pick the first availability zone in the region for our public and the second availability zone in the region for our private subnet. Copy the following code into your YAML file to show all the availability zones that are present in the region, and which ones will be used for the public and private subnets.

Copy the following code to create the public subnet in the first availability zone in us-east-1 region. Do note that we are provisioning our public subnet with CIDR range 172.32.1.0/24

Copy the following code to deploy the private subnet in the second availability zone in us-east-1 region. It will use the CIDR range 172.32.2.0/24

Hold on! To make a public subnet, it is not enough to just create a subnet. We need to create routes from that subnet to the internet gateway! The below code will address this. The private subnet does not need any such routes, it will use the default route table.

As planned, we will be deploying jumphosts within the public subnet. By default, you won’t be able to externally connect to the EC2 instances deployed within the public subnet because the default security group does not allow this.

To remediate this, we will create a new security group that will allow RDP access and assign it to the jumphost server. For simplicity, the security group will allow RDP access from anywhere, however please ensure that for your environment, you have locked it down to a few external IP addresses.

Phew! Finally, we are ready to deploy our jumphost! Copy the following code for this

I would like to point out a few things

  • The jumphost is running on a t2.micro instance. This instance type is usually sufficient for a jumphost in a lab environment, however if you need more performance, this can be changed (changing the instance type from t2.micro can take you over the AWS free tier limits and subsequently add to your monthly costs)
  • The image parameter refers to the AMI ID of the Windows 2016 base image that is currently available within the AWS console. AWS, from time to time, changes the images that are available. Please check within the AWS console to ensure that the AMI ID is valid before running the playbook
  • Instance tags are tags that are attached to the instance. In this case, the instance tags have been used to name the jumphost win2016jh.

Important Information

The following parameters are extremely important, if you do not intend on deploying a new EC2 instance for the same server every time you re-run this Ansible playbook.

exact_count – this parameter specifies the number of EC2 instances of a server that should be running whenever the Ansible playbook is run. If the current number of instances doesn’t match this number, Ansible either creates new EC2 instances for this server or terminates the extra EC2 instances. The servers are identified using the count_tag

count_tag – this is the instance tag that is used to identify a server. Multiple instances of the same server will have the same tag applied to them. This allows Ansible to easily count how many instances of a server are currently running.

Next, we will deploy the servers within the private subnet. Wait a minute! By default, the servers within the private subnet will be assigned the default security group. The default security group allows unrestricted access to all EC2 instances that have been attached to the default security group. However, since the jumphost is not part of this security group, it will not be able to connect to the servers in the private subnet!

Let’s remediate this issue by creating a new security group that will allow RDP access from the public subnet to the servers within the private subnet (in a real environment, this should be restricted further, so that the incoming connections are from particular servers within the public subnet, and not from the whole subnet itself). This new security group will be associated with the servers within the private subnet.

Copy the following code into your YAML file.

We are now at the end of the YAML file. Copy the code below to provision the windows 2016 server within the private subnet (the server will be tagged with name=win2016svr)

Save the playbook with a meaningful name. I named my playbook Ansible-create-AWS-environment.yml

The full Ansible playbook can be downloaded from https://gist.github.com/nivleshc/344dca91e3d0349c8a359b03853886be

Running the Ansible Playbook

Before we run the playbook, we need to tell Ansible about all the computers that are within the management scope. This is done using an inventory file, which contains a group name within square brackets eg [webservers] and below that, all the computers that will be in that group. Then in the playbook, we just target the group, which in turn targets all the computers in that group.

However, in our scenario, we are directly targeting the local computer (refer to the second line in the YAML file that shows hosts: localhost). In this regard, we can get away with not providing an inventory file. However, do note that doing so will mean that we can’t use anything other than localhost to reference a computer within our playbook.

Let’s create an inventory file called hosts in the same folder as where the playbook is saved. The contents of the file will be as listed below.

[local]
localhost

We are ready to run the playbook now.

Open a terminal session and change to the folder where the playbook was saved.

We need to create some environment variables to store the user details that Ansible will use to connect to AWS. This is where the access key and secret access key that we created initially will be used. Run the following command

export AWS_ACCESS_KEY_ID={access key id}
export AWS_SECRET_ACCESS_KEY={secret access key}

Now run the playbook using the following command (as previously mentioned, we could get away with not specifying the inventory file, however this means that we only can use localhost within the playbook)

ansible-playbook -i hosts ansible-create-aws-environment.yml

You should now see each of the tasks being executed, with the output being shown (remember that after each task, we have a follow-up task that shows the output using the debug keyword? )

Once the playbook execution has completed, check your AWS console to confirm that the following items have been created within the us-east-1 (North Virginia) region

  • A VPC called ansibleVPC with the CIDR 172.32.0.0/16
  • An internet gateway called ansibleVPC_igw
  • A public subnet in the first availability zone with CIDR 172.32.1.0/24
  • A private subnet in the second availability zone with CIDR 172.32.2.0/24
  • A route table called rt_ansibleVPC_PublicSubnet
  • A security group for jumphosts called sg_ansibleVPC_publicsubnet_jumphost
  • A security group for the servers in private subnet called sg_ansibleVPC_privatesubnet_servers
  • An EC2 instance in the public subnet representing a jumphost named win2016jh
  • An EC2 instance in the private subnet representing a server named win2016svr

Once the provisioning is complete, to test, connect to the jumphost and then from there connect to the server within the private subnet.

Don’t forget to turn off the EC2 instances if you don’t intend on using them

Closing Remarks

Ansible is a great automation tool and can be used to both provision and manage infrastructure within AWS.

Having said that, I couldn’t find an easy way to do post provisioning tasks (eg assigning roles, installing additional packages etc) after the server has been provisioned, without getting Ansible to connect directly to the provisioned server. This can be a challenge if the Ansible control machine is external to AWS and the provisioned server is within an AWS private subnet. With AWS CloudFormation, this is easily done. If anyone has any advice on this, I would appreciate it if you can leave it in the comments below.

I will surely be using Ansible for most of my automations from now on.

Till the next time, enjoy!

Using AWS EC2 Instances to train a Convolutional Neural Network to identify Cows and Horses

Background

Machine Learning (ML) and Artificial Intelligence (AI) has been a hobby of mine for years now. After playing with it approximately 8 years back, I let it lapse till early this year, and boy oh boy, how things have matured! There are products in the market these days that use some form of ML – some examples are Apple’s Siri, Google Assistant, Amazon Alexa.

Computational power has increased to the point where calcuations that took months can now be done within days. However, the biggest change has come about due to the vast amounts of data that the models can be trained on. More data means better accuracy in models.

If you have taken any programming course, you would remember the hello world program. This is a foundation program, which introduces you to the language and gives you the confidence to continue on. The hello world for ML is identifying cats and dogs. Almost every online course I have taken, this is the first project that you build.

For anyone wanting a background on Machine Learning, I would highly recommend Andrew Ng’s https://www.coursera.org/learn/machine-learning in Coursera. However, be warned, it has a lot of maths 🙂 If you are able to get through it, you will get a very good foundational knowledge on ML.

If theory is not your cup of tea, another way to approach ML is to just implement it and learn as you go. You don’t need to get a PhD in ML to start implementing it. This is the philosophy behind Jeremy Howard’s and Rachel Thomas’s http://www.fast.ai. They take you through the implementation steps and introduce you to the theory on a need to know basis, in essence you are doing a top down approach.

I am still a few lessons away from finishing the fast.ai course however, I have learnt so much and I cannot recommend it enough.

In this blog, I will take you through the steps to implement a Convolutional Neural Network (CNN) that will be able to pick out horses from cows. CNNs are quite complicated in nature so we won’t go into the nitty-gritty details on creating them from scratch. Instead, we will use the foundational libraries from fast.ai’s lesson 1 and modify it abit, so that instead of identifying cats and dogs, we will use it to identify cows and horses.

In the process, I will introduce you to a tool that will help you scrape Google for your own image dataset.

Most important of all, I will show you how the amount of data used to train your CNN model affects its accuracy.

So, put your seatbelts on and lets get started!

 

1. Setting up the AWS EC2 Instance

ML requires a lot of processing power. To get really good throughput, it is recommended to use GPUs instead of CPUs. If you were to build a kit to try this at home, it can easily cost you a few thousands of dollars, not to mention the bill for the cooling and electricity usage.

However, with Cloud Computing, we don’t need to go out and buy the whole kit, instead we can just rent it for as long as we want. This provides a much affordable way to learn ML.

In this blog, we will be using AWS EC2 instances. For the cheapest GPU cores, we will use a p2.xlarge instance. Be warned, these cost $0.90/hr, so I would suggest turning them off after using them, otherwise you will surely rack up a huge bill.

Reshma has done a fantastic job of putting together the instructions on setting up an AWS Instance for running fast.ai course lessons. I will be using her instructions, with a few modifications. Reshma’s instructions can be found here.

Ok lets begin.

  • Login to your AWS Console
  • Go to the EC2 section
  • On the top left menu, you will see EC2 Dashboard. Click on Limits under it
  • AWS_Dashboard_EC2_Limits
  • Now, on the right you will see all the type of EC2 instances you are allowed to run. Search for p2.xlarge instances. These have a current limit of zero, meaning you cannot launch them. Click on Request limit increase and then fill out the form to justify why you want a p2.xlarge instance. Once done, click on Submit. In my case, within a few minutes, I received an email saying that my limit increase had been approved.
  • Click on EC2 Dashboard from the left menu
  • Click on Launch Instance
  • In the next screen, in the left hand side menu, click on Community AMIs
  • On the right side of the screen, search for fast.ai
  • From the results, select fastai-part1v2-p2
  • In the next screen (Instance Type) filter by GPU compute and choose p2.xlarge
  • In the next screen configure the instance details. Ensure you get a public IP address (Auto-assign Pubic IP) because you will be connecting to this instance over the internet. Once done, click Next: Add Storage
  • In the next screen, you don’t need to do anything. Just be aware that the community AMI comes with a 80GB harddisk (at $0.10/GB/Month, this will amount to $8/Month). Click Next
  • In the next screen, add any tags for the EC2 Instance. To give the instance a name, you can set the Key to Name and the Value to fastai. Click Next
  • For security groups, all you need to do is allow SSH to the instance. You can leave the source as 0.0.0.0/0 (this allows connections to the EC2 instance from any public IP address). However, if you want to be super secure, you can set the source to your current ip address. However, doing this means that should your public ip address change (hardly any ISPs give you a static IP address, unless you pay extra), you will have to go back into the AWS Console and update the source in the security group. Click Next
  • In the next section, check that all details are correct and then click on Launch. You will be asked for your key pair. You can either choose an existing key pair or create a new one. Ensure you keep the key pair in a safe place because whoever possesses it can connect to your EC2 instance.
  • Now, sit back and relax, Within a few minutes, your EC2 instance will be ready. You can monitor the progress in the EC2 Dashboard

DON’T FORGET TO SHUTDOWN THE INSTANCE WHEN NOT USING IT. AT $0.90/hr, IT MIGHT NOT SEEM MUCH, HOWEVER THE COST CAN EASILY ACCUMULATE TO SOMETHING QUITE EXPENSIVE

2. Creating the dataset

To train our Convolutional Neural Network (CNN), we need to get lots of images of cows and horses. This got me thinking. Why not get it off Google? But, then this provided another challenge. How do I download all the images? Surely I don’t want to be sitting there right clicking each search result and saving it!

After some googling, I landed on https://github.com/hardikvasa/google-images-download. It does exactly as to what I wanted. It will do a google image search using a keyword and download the results.

Install it using the instructions provided in the link above. By default, it only downloads 100 images. As CNNs need lots more, I would suggest installing chromedriver. The instructions to do this is in the Troubleshooting section under ## Installing the chromedriver (with Selenium)

To download 1000 images of cows and horses, use the following command line (for some reason the tool only downloads around 800 images)

  • the downloaded images will be stored in the subfolder cows/downloaded and horses/downloaded in the /Users/x/Documents/images folder.
  • keyword denotes what we are searching for in google. For cows, we will use cow because we want a single cow’s photo. The same for horses.
  • –chromedriver provides the path to where the chromedriver has been stored
  • the images will be in jpg format
googleimagesdownload --keywords "cow" --format jpg --output_directory "/Users/x/Documents/images/" --image_directory "cows/downloaded" --limit 1000 --chromedriver /Users/x/Documents/tools/chromedriver
googleimagesdownload --keywords "horse" --format jpg --output_directory "/Users/x/Documents/images/" --image_directory "horses/downloaded" --limit 1000 --chromedriver /Users/x/Documents/tools/chromedriver

3. Finding and Removing Corrupt Images

One disadvantage of using googleimagedownload script is that, at times a downloaded image cannot be opened. This will cause issues when our CNN tried to use it for training/validating.  To ensure our CNN does not encounter any issues, we will do some housekeeping before hand and remove all corrupt images (images that cannot be opened).

I wrote the following python script to find and move the corrupt images to a separate folder. The script uses the matplotlib library (the same library used by the fast.ai CNN framework) If you don’t have it, you will need to download it from https://matplotlib.org/users/installing.html.

The script assumes that within the root folder, there is a subfolder called downloaded which contains all the images. It also assumes there is a subfolder called corrupt within the root folder. This is where the corrupt images will be moved to. Set the root_folder_path to the parent folder of the folder where the images are stored.

#this script will go through the downloaded images and find those that cannot be opened. These will be moved to the corrupt folder.

#load libraries
import matplotlib.pyplot as plt
import os

#image folder
root_folder_path = '/Users/x/Documents/images/cows/'
image_folder_path = root_folder_path + 'downloaded/'
corrupt_folder_path = root_folder_path + 'corrupt' #folder were the corrupt images will be moved to

#get a list of all files in the img folder
image_files = os.listdir(f'{image_folder_path}')

print (f'Total Image Files Found: {len(image_files)}')
num_image_moved = 0

#lets go through each image file and see if we can read it
for imageFile in image_files:
 filePath = image_folder_path + imageFile
 #print(f'Reading {filePath}')
 try:
 valid_img = plt.imread(f'{filePath}')
 except:
 print (f'Error reading {filePath}. File will be moved to corrupt folder')
 os.rename(filePath,os.path.join(corrupt_folder_path,imageFile))
 num_image_moved += 1

print (f'Moved {num_image_moved} images to corrupt folder')

For some unknown reason, the script, at times, moves good images into the corrupt folder as well. I would suggest that you go through the corrupt images and see if you can open them (there won’t be many in the corrupt folder). If you can, just manually move them back into the downloaded folder.

To make the images easier to handle, lets rename them using the following format.

  • For the images in the cows/downloaded folder rename them to a format CowXXX.jpg where XXX is a number starting from 1
  • For the images in the horses/downloaded folder rename them to a format HorseXXX.jpg where XXX is a number starting from 1

 

4. Transferring the images to the AWS EC2 Instance

In the following sections, I am using ssh and scp which come builtin with MacOS. For Windows, you can use putty for ssh and WinSCP for scp

A CNN (or any other Neural Network model) is trained using a set of images. Once training has finished, to find how accurate the model is, we give it a set of validation images (these are different to those it was trained on, however we know what these images are of) and ask it to identify the images. We then compare the results with what the actual image was, to find the accuracy.

 

In this blog, we will first train our CNN on a small set of images.

Do the following

  • create a subfolder inside the cows folder and name it train
  • create a subfolder inside the cows folder and name it valid
  • move 100 images from the cows/downloaded folder into the cows/train folder
  • move 20 images from the cows/downloaded folder into the cows/valid folder

Make sure the images in the cows/train folder are not the same as those in cows/valid folder

Do the same for the horses images, so basically

  • create a subfolder inside the horses folder and name it train
  • create a subfolder inside the horses folder and name it valid
  • move 100 images from the horses/downloaded folder into the horses/train folder
  • move 20 images from the horses/downloaded folder into the horses/valid folder

Now connect to the AWS EC2 instance the following command line

ssh -i key.pem ubuntu@public-ip

where

  • key.pem is the key pair that was used to create the AWS EC2 instance (if the key pair is not in the current folder then provide the full path to it)
  • public-ip is the public ip address for your AWS EC2 instance (this can be obtained from the EC2 Dashboard)

Once connected, use the following commands to create the required folders

cd data
mkdir cowshorses
mkdir cowhorses/train
mkdir cowhorses/valid
mkdir cowhorses/train/cows
mkdir cowhorses/train/horses
mkdir cowhorses/valid/cows
mkdir cowhorses/valid/horses

Close your ssh session by typing exit

Run the following commands to transfer the images from your local computer to the AWS EC2 instance

To transfer the cows training set
scp -i key.pem /Users/x/Documents/images/cows/train/*  ubuntu@public-ip::~/data/cowshorses/train/cows

To transfer the horses training set
scp -i key.pem /Users/x/Documents/images/horses/train/*  ubuntu@public-ip::~/data/cowshorses/train/horses

To transfer the cows validation set
scp -i key.pem /Users/x/Documents/images/cows/valid/*  ubuntu@public-ip::~/data/cowshorses/valid/cows

To transfer the horses validation set
scp -i key.pem /Users/x/Documents/images/horses/valid/*  ubuntu@public-ip::~/data/cowshorses/valid/horses

5. Starting the Jupyter Notebook

Jupyter Notebooks are one of the most popular tools used by ML and data scientists. For those that aren’t familiar with Jupyter Notebooks, in a nutshell, it a web page that contains descriptions and interactive code. The user can run the code live from within the document. This is possible because Jupyter Notebook’s execute the code on the server it is running on and then displays the result in the web page. For more information, you can check out http://jupyter.org

In our case, we will be running the Jupyter Notebook on the AWS EC2 instance. However, we will be accessing it through our local computer. For security reasons, we will not publish our Jupyter Notebook to the whole wide world (lol that does spell www).

Instead, we will use the following ssh command to bind our local computer’s tcp port 8888 to the AWS EC2 instance’s tcp port 8888 (this is the port on which the Jupyter Notebook will be running) when we connect to it. This will allow us to access the Jupyter Notebook as if it is running locally on our computer, however the connection will be tunnelled to the AWS EC2 instance.

ssh  -i key.pem ubuntu@public-ip -L8888:localhost:8888

Next, run the following commands to start an instance of Jupyter Notebook

cd fastai
jupyter notebook

After the Jupyter Notebook starts, it will provide a URL to access it, along with the token to authenticate with. Copy it and then paste it into a browser on your local computer.

You will now be able to access the fastai Jupyter Notebook.

Follow the steps below to open Lesson 1.

  • click on the courses folder
  • once inside the courses folder,  click on the  dl1 folder

In the next screen, find the file lesson1.ipynb and double-click it. This will launch the lesson1 Jupyter Notebook in another tab.

Give yourself a big round of applause for reaching so far!

Now, start from the top of lesson1 and go through the first three code sections and execute them. To execute the code, put the mouse pointer in the code section and then press Shift+Enter.

In the next section, change the path to where we moved the cows and horses pictures to. It should look like below

PATH = "data/cowshorses/"

Then, execute this code section.

Skip the following sections

  • Extra steps if NOT using Crestle or Paperspace or our scripts
  • Extra steps if using Crestle

Just a word of caution. The original Jupyter Notebook is meant to distinguish between cats and dogs. However, since we are using it to distinguish between cows and horses, whenever you see a mention of cats, change it to cows and whenever you see a mention of dogs, change it to horses.

The following lines don’t need any changing, so just execute them as they are

os.listdir(PATH)
os.listdir(f'{PATH}valid')

In the next line, replace cats with cows so that you end up with the following

files = !ls {PATH}valid/cows | head
files

Execute the above code. A list of the first 10 cow image files will be displayed.

Next, lets see what the first cow image looks like.

In the next line, change cats to cows to get the following.

img = plt.imread(f'{PATH}valid/cows/{files[0]}')
plt.imshow(img);

Execute the code and you will see the cow image displayed.

Execute the next two code sections. Leave the section after that commented out.

Now, instead of creating a CNN model from scratch, we will use one that was pre-trained on ImageNet which had 1.2 million images and 1000 classes. So it already knows quite a lot about how to distinguish objects. To make it suitable to what we want to do, we will now train it further on our images of cows and horses.

The following defines which model to use and provides the data to train on (the CNN model that we will be using is called resnet34). Execute the below code section.

data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(resnet34, sz))
learn = ConvLearner.pretrained(resnet34, data, precompute=True)

And now for the best part! Lets train the model and give it a learning rate of 0.01.

learn.fit(0.01, 1)

After you execute the above code, the model will be trained on the cows and horses images that were provided in the train folders. The model will then be tested for accuracy by getting it to identify the images contained in the valid folders. Since we already know what the images are of, we can use this to calculate the model’s accuracy.

When I ran the above code, I got an accuracy of 0.75. This is quite good since it means the model can identify cows from horses 75% of the time. Not to forget, we used only 100 cows and 100 horses images to train it, and it didn’t even take that long to train it !

Now, lets see what happens when we give it loads more images to train on.

BTW to get more insights into the results from the trained model,  you can go through all the sections between the lines learning.fit(0.01,1) and Choosing a learning rate.

Another take at training the model

From all the literature I have been reading, one point keeps on repeating. More data means better models. Lets put this to the test.

This time around we will give the model ALL the images we downloaded.

Do the following.

  • on your local computer, move the photos back to the downloaded folder
    • move photos from cows/train to cows/downloaded
    • move photos from cows/valid to cows/downloaded
    • move photos from horses/train to horses/downloaded
    • move photos from horses/valid to horses/downloaded
  • on your local computer, move 100 photos of cows to cows/valid folder and the rest to the cows/train folder
    • move 100 photos from cows/downloaded to cows/valid folder
    • move the rest of the photos from cows/downloaded to cows/train folder
  • on your local computer, move 100 photos for horses to horses/valid and the rest to horses/train folder
    • move 100 photos from horses/downloaded to horses/valid folder
    • move the rest of the photos from horses/downloaded to horses/train folder
  • on the AWS EC2 instance, delete all the photos under the following folders
    • /data/cowshorses/train/cows
    • /data/cowshorses/train/horses
    • /data/cowshorses/valid/cows
    • /data/cowshorses/valid/horses

Use the following commands to copy the images from the local computer to the AWS EC2 Instance

To transfer the cows training set
scp -i key.pem /Users/x/Documents/images/cows/train/*  ubuntu@public-ip::~/data/cowshorses/train/cows

To transfer the horses training set
scp -i key.pem /Users/x/Documents/images/horses/train/*  ubuntu@public-ip::~/data/cowshorses/train/horses

To transfer the cows validation set
scp -i key.pem /Users/x/Documents/images/cows/valid/*  ubuntu@public-ip::~/data/cowshorses/valid/cows

To transfer the horses validation set
scp -i key.pem /Users/x/Documents/images/horses/valid/*  ubuntu@public-ip::~/data/cowshorses/valid/horses

Now that everything has been prepared, re-run the Jupyter Notebook, as stated under Starting Jupyter Notebook above (ensure you start from the top of the Notebook).

When I trained the model on ALL the images (less those in the valid folder) I got an accuracy of 0.95 ! Wow that is soo amazing! I didn’t do anything other than increase the amount of images in the training set.

Final thoughts

In a future blog post, I will show you how you can use the trained model to identify cows and horses from a unlabelled set of photos.

For now, I would highly recommend that you use the above mentioned image downloader to scrape Google for some other datasets. Then use the above instructions to train the model on those images and see what kind of accuracy you can achieve (maybe try identifying chickens and ducks?)

As mentioned before, once finished, don’t forget to shut down your AWS EC2 instance. If you don’t need it anymore, you can terminate it, to save on storage costs as well.

If you are keen about ML, you can check out the courses at http://www.fast.ai (they are free)

If you want to dabble in the maths behind ML, as perviously mentioned, Andrew Ng’s https://www.coursera.org/learn/machine-learning is one of the finest.

Lastly, if you are keen to take on some ML challenges, check out https://www.kaggle.com They have lots and lots competitions running all the time, some of which pay out actual money. There are lots of resources as well and you can learn off others on the site.

Till the next time, Enjoy 😉

Deploying an Active Directory Forest using AWS CloudFormation

Introduction

Wow, it is amazing how time flies. Almost two years ago, I wrote a set of blogs that showed how one can use Azure Resource Manager (ARM) templates and Desired State Configuration (DSC) scripts to deploy an Active Directory Forest automatically.

For those that would like to take a trip down memory lane, here is the link to the blog.

Recently, I have been playing with AWS CloudFormation and I am simply in awe by its power. For those that are not familiar with AWS CloudFormation, it is a tool, similar to Azure Resource Manager, that allows you to “code” your computing infrastructure in Amazon Web Services. Long gone are the days when you would have to sit down, pressing each button and choosing each option to deploy your environment. Cloud computing provides you with a way to interface with the fabric, so that you can script the build of your environment. The benefits of this are enormous. Firstly, it allows you to standardise all your builds. Secondly, it allows you to have a live as-built document (the code is the as-built document). Thirdly, the code is re-useable. Most important of all, since the deployment is now scripted, you can automate it.

In this blog I will show you how to create an AWS CloudFormation template to deploy an AWS Elastic Compute Cloud (EC2) Windows Server instance. The template will also include steps to promote the EC2 instance to a Domain Controller in a new Active Directory Forest.

Guess what the best part is? Once the template has been created, all you will have to do is to load it into AWS CloudFormation, provide a few values and sit back and relax. AWS CloudFormation will do everything for you from there on!

Sounds interesting? Lets begin.

Creating the CloudFormation Template

A CloudFormation template starts with a definition of the parameters that will be used. The person running the template (lets refer to them as an operator) will be asked to provide the values for these parameters.

When defining a parameter, you will provide the following

  • a name for the parameter
  • its type
  • a brief description for the parameter so that the operator knows what it will be used for
  • any constraints you want to put on the parameter, for instance
    • a maximum length (for strings)
    • a list of allowed values (in this case a drop down list is presented to the operator, to choose from)
  • a default value for the parameter

For our template, we will use the following parameters.

Next, we will define some mappings. Mappings allow us to define the values for variables, based on what value was provided for a parameter.

When creating EC2 instances, we need to provide a value for the Amazon Machine Image (AMI) to be used. In our case, we will use the OS version to decide which AMI to use.

To find the subnet into which the EC2 instance will be deployed in, we will use the Environment and AvailabilityZone parameters to find it.

The code below defines the mappings we will use

The next section in the CloudFormation template is Resources. This defines all resources that will be created.

If you have any experience deploying Active Directory Forests, you will know that it is extremely simple to do it using PowerShell scripts. Guess what, we will be using PowerShell scripts as well 😉 Now, after the EC2 instance has been created, we need to provide the PowerShell scripts to it, so that it can run them. We will use AWS Simple Storage Service (S3) buckets to store our PowerShell scripts.

To ensure our PowerShell scripts are stored securely, we will allow access to it only via a certain role and policy.

The code below will create an AWS Identity and Access Management (IAM) role and policy to access the S3 Bucket where the PowerShell scripts are stored.

We will use cf-init to do all the heavy lifting for us, once the EC2 instance has been created. cf-init is a utility that is present by default in EC2 instances and we can ask it to perform tasks for us.

To trigger cf-init, we will use the Userdata feature of EC2 instance provisioning. cf-init, when started, will check the EC2 Metadata for the credentials it will use, and it will also check it for all the tasks it needs to perform.

Below is the metadata that will be used. For simplicity, I have hardcoded the URL to the files in the S3 bucket.

As you can see, I have first defined the role that cf-init will use to access the S3 bucket. Next, the following tasks will be carried out, in the order defined in the configuration set

  • get-files
    • it will download the files from S3 and place them in the local directory c:\s3-downloads\scripts.
  • configure-instance (the commands in this section are run in alphabetical order, that is why I have prefixed them with a number, to ensure it follows the order I want)
    • It will change the execution policy for PowerShell to unrestricted (please note that this is just for demonstration purposes and the execution policy should not be made this relaxed).
    • next, the name of the server will be changed to what was provided in the Parameters section
    • the following Windows Components will be installed (as defined in the Add-WindowsComponents.ps1 script file)
      • RSAT-AD-PowerShell
      • AD-Domain-Services
      • DNS
      • GPMC
    • the Active Directory Forest will be created, using the Configure-ADForest.ps1 script and the values provided in the Parameters section

In the last part of the CloudFormation template, we will provide the UserData information that will trigger cfn-init to run and do all the configuration. We will also tag the the EC2 instance, based on values from the Parameters section.

For simplicity, I have hardcoded the security group that will be attached to the EC2 instance (this is defined as GroupSet under NetworkInterfaces). You can easily create an additional parameter for this, if you want.

Finally, our template will output the instance’s hostname, environment it has been created in and its privateip. This provides an easy way to identify the EC2 instance once it has been created.

Below is the last part of the template

Now all you have to do is login to AWS CloudFormation, load the template we have created, provide the parameter values and sit back and relax.

AWS CloudFormation will take it from here and do everything for you 😉

How easy was that? Magic 🙂

The complete CloudFormation template is available at https://gist.github.com/nivleshc/867b1a2ca119c7d22cf215b5a9a5de02

The two PowerShell Scripts that are used in the CloudFormation template can be downloaded using the links below

Add-WindowsComponents.ps1

Configure-ADForest.ps1

For anyone deploying an Active Directory Forest in AWS, I hope the above comes in handy.

Enjoy 😉

Amazon QuickSight – An elegant and easy to use business analytics tool

Introduction

Recently, I had a requirement for a tool to visualise some data I had collected. My requirements were very simple. I didn’t want something that would cost me a lot, and at the same time I wanted the reports to be elegant and informative. Most of all, I didn’t want to have to go through pages and pages of documentation to learn how to use it.

As my data was within Amazon Web Services (AWS), I thought to check if AWS had any such offerings. Guess what, there was indeed a tool just for what I wanted, and after using it, I was amazed at how simple and elegant it is.

In this blog, I will show how you can easily get started with Amazon QuickSight. I will take you through the steps to import your data into Amazon QuickSight and then create some informative visualisations.

Some background on Amazon QuickSight

Pricing

Amazon QuickSight is very inexpensive, infact, if your data is not too much, you won’t have to pay anything!

For standard edition use, Amazon QuickSight provides 1GB of SPICE for the first user free per month. SPICE is an acronym for Super-fast, Parallel, In-memory, Calculation Engine and it uses a combination of columnar storage, in-memory technologies enabled through the latest hardware innovations, machine code generation, and data compression to allow users to run interactive queries on large datasets and get rapid responses.  SPICE is the calculation engine that Amazon QuickSight uses.

Any additional SPICE is priced at $USD0.25 per GB/month. For the latest pricing, please refer to https://aws.amazon.com/quicksight/#Pricing

Data Sources

Currently Amazon QuickSight supports the following data sources

  • Relational Data Sources
    • Amazon Athena
    • Amazon Aurora
    • Amazon Redshift
    • Amazon Redshift Spectrum
    • Amazon S3
    • Amazon S3 Analytics
    • Apache Spark 2.0 or later
    • Microsoft SQL Server 2012 or later
    • MySQL 5.1 or later
    • PostgreSQL 9.3.1 or later
    • Presto 0.167 or later
    • Snowflake
    • Teradata 14.0 or later
  • File Data Sources
    • CSV/TSV – (comma separated, tab separated value text files)
    • ELF/CLF – Extended and common log format files
    • JSON – Flat or semi-structured data files
    • XLSX – Microsoft Excel files

Unfortunately, currently Amazon DynamoDB is not supported as a native data source. Since my data is in Amazon DynamoDB, I had to write some custom lambda functions to export it to a csv file, so that it could be imported into Amazon QuickSight.

Ok, time for that walk-through I promised earlier.  For this blog, I will be using an S3 bucket as my data source. It will contain the CSV files that I will use for analysis in Amazon QuickSight.

Step 1 – Create S3 buckets

If you haven’t already done so, create an S3 bucket that will contain the csv files. The S3 bucket does not have to be publicly accessible. Once created, upload the csv files into the S3 bucket.

In my case, the csv file is called orders.csv and its location is https://s3.amazonaws.com/sample/orders.csv (to get the URL to your S3 file, login to the S3 console and navigate to the S3 bucket that contains the file. Click the S3 bucket to open it, then click the file name to open its properties. Under Overview you will see Link. This is the URL to the file)

Step 2 – Create an Amazon QuickSight Account

Before you start using Amazon QuickSight, you must create an account. Unfortunately, I couldn’t find a way for creating an Amazon QuickSight account without creating an Amazon AWS account. If you don’t have an existing Amazon AWS account, you can create an AWS Free Tier account. Once you have got an AWS account, go ahead and create an Amazon QuickSight account at https://aws.amazon.com/quicksight/.

While creating your Amazon QuickSight account, you will be asked if you would like Amazon QuickSight to auto-discover your Amazon S3 buckets. Enable this and then click to Choose S3 buckets. Choose the S3 bucket that you created in Step 1 above. This will give Amazon QuickSight read-only access to the S3 bucket, so that it can read the data for analysis.

Step 3 – Create a manifest file

A manifest file is a JSON file that provides the location and format of the data files to Amazon QuickSight. This is required when creating a data set for S3 data sources. Please refer to https://docs.aws.amazon.com/quicksight/latest/user/supported-manifest-file-format.html if you would like more information about manifest files.

Below is my manifest file, which I have affectionately named ordersmanifest.json.

{
   "fileLocations": [
      {
         "URIs": [
            "https://s3.amazonaws.com/sample/orders.csv"
         ]
      },
   ],
   "globalUploadSettings": {
      "format": "CSV",
      "delimiter": ",",
      "textqualifier": "'",
      "containsHeader": "true"
   }
}

Once created, upload the manifest file into the same S3 bucket as to where the csv file is stored.

Step 4 – Create a data set

  • Login to your Amazon QuickSight account. From the top right, click on Manage data
  • In the next screen, click on New data set
  • In the next screen, for Create a Data Set FROM NEW DATA SOURCES, click on S3
  • In the next screen
    • provide a name for the data source
    • for Upload a manifest file ensure URL is clicked and enter the URL to the manifest file (you can get the url by logging into the S3 console, and then clicking on the manifest file to reveal its properties. Under the Overview tab, you will see Link. This is the URL to the manifest file).NewS3DataSource
    • Click Connect
    • Amazon QuickSight will now read the manifest file and then import the csv file to SPICE. You will see the following screenFinishDataSetCreation
    • Click on Edit/Preview data.
    • In the next screen, you will see the contents of the data file that was imported, along with the Fields name on the left. If you want to exclude any columns from the analysis, simply untick them (I unticked orderTime (S) since I didn’t need it) EditPreviewDataSet
    • By default, the data is called Group 1. To customise the name, replace Group 1 with a text of your choice (I have renamed my data to Orders Data)RenameGroup1Label
    • Click Save & visualize from the top menu

Step 5 – Create Visualisations

Now that you have imported the data into SPICE, you can start analysing it and creating visualisations.

After step 4, you should be in the Analysis section.

  • Depending on which visualisation you want, you can select the respective type under Visual types from the bottom left hand side of the screen. For my visualisations, I chose Pie Chart (side note – you will notice that orderTime (S)  isn’t listed under Fields list. This is because we had unticked it in the previous screen)OrdersDataAnalysis-01
  • I want to create two Pie Charts, one to show me analysis about what is the most popular foodName and another to find out what is the most popular drinkName. For the first Pie Chart, drag foodName (S) from the Fields list to the Value – Add a measure here box  in the top of the screen. Then drag foodName (S) from the Fields list to the Group/Color – Add a dimension here box in the top of the screen. You will see the followingOrdersDataAnalysis-02
  • You can customise the visualisation title Count of Foodname (S) by Foodname (S) by clicking it and then changing the text (I have changed the title to Popularity of Food Types)FoodNamePopularity
  • If you look closely, the legend on the right hand side doesn’t serve much purpose since the pie slices are already labelled quite well. You can also get rid of the legend and get more space for your visual. To do this, click on the down arrow above FoodName (S) on the right and then select Hide legend FoodNameHideLengend
  • Next, lets create a Pie Chart visualisation for drinkName. From the top menu, click on Add and then Add visual drinkNameAddVisual
  • You will now have another Canvas at the bottom of the first Pie Chart. Click this new canvas area to select it (a blue border will appear to show that it is selected). From Visual types at the bottom left hand side, click on the Pie Chart visual. Then from the top, click on Field wells to expose the Value and Group/Color boxes for the second canvas drinkNameCanvas
  • From the Field list on the left, drag drinkName (S) to the Value – Add a measure here box  in the top of the screen. Then drag drinkName (S) from the Fields list to the Group/Color – Add a dimension here box in the top of the screen. You will now see the following foodanddrinkvisual
  • We are almost done. I actually want the two Pie Charts to sit side by side, instead of one ontop ofthe other. To do this, I will show you a neat trick. In each of the visuals, at the bottom right border, you will see two diagonal lines. If you move your mouse pointer over them, they change to a resizing cursor. Use this to resize the visual’s canvas area. Also, in the middle of the top border of the visual, you will see two rows of gray dots. Click your mouse pointer on this and drag to the location you want to move the visual to.VisualResizeandMove
  • I have hidden the legend for the second visual, customised the title and resized both the visuals and moved them side by side. Viola! Below is what I get. Not bad aye!BothVisualsSidebySide

Step 6 – Create a dashboard

Now that the visuals have been created, they can be shared it with others. This can be done by creating a dashboard. A dashboard is a read-only snapshot of the analysis. When you share the dashboard with others, they can view and filter the dashboard data, however any filters applied to the dashboard visual exist only when the user is viewing the dashboard, and aren’t saved once it is closed.

One thing to note about sharing dashboards – you can only share dashboards with users who have an Amazon QuickSight account.

Creating a dashboard is very easy.

  • In the Analysis screen, on the top right corner, click on Share and then select Create dashboardCreateDashboard
  • You can either replace an existing dashboard or create a new one. In our case, since we are creating a new dashboard, select Create a new dashboard as and enter a name for the dashboard. Once finished, click Create dashboardCreateDashboard-Name
  • You will then be asked to enter the username or email address of those you want to share the dashboard with. Enter this and click on Share ShareDashboard
  • That’s it, your dashboard is now created. To access it, go to the Amazon QuickSight home screen (click on the Amazon QuickSight icon on the top left hand side of the screen) and then click on All dashboards. Those that you have shared the dashboard with will also be able to see it once they login to their Amazon QuickSight account.AllDashboards

Step 6 – Refreshing the Data Set

If your data set continually changes, your visualisations/dashboards will not show the updated information. This can be done by refreshing the data set. Doing this will import the new data into SPICE, which will then automatically update the analysis/visualisations and dashboards

Note: you will have to manually reload the webpage to see the updated visualisations and dashboard

There are two ways of refreshing data sets. One is to do it manually while the other is to use a schedule. The scheduled data refresh allows for the data to be automatically refreshed at a certain time daily, weekly or monthly. A maximum of five scheduled refreshes can be configured.

The steps below show how you can manually refresh the data or create schedules to refresh the data

  • From the Amazon QuickSight main screen, click on Manage data from the top left of the screen ManageData
  • In the next screen, you will see all your currently configured data sets. Click the Orders Data dataset (this is the one we had created previously).
  • In the next screen, you will see Refresh Now and Schedule refreshManualScheduleDataRefresh
  • Clicking on Refresh Now will manually refresh the data. Clicking on Schedule refresh will bring up the screen where you can configure a schedule for refreshing the data automatically.

 

That’s it folks! Wasn’t that simple? If you already have an Amazon AWS account, I would strongly recommend giving Amazon QuickSight a try for all your analytics needs. Even if you don’t have an Amazon AWS account, I would still suggest getting an AWS free tier account to try it out.

Enjoy 😉

 

My path to AWS Certified Solutions Architect – Associate

Almost a week and a half ago, I sat for and passed my AWS Certified Solutions Architect – Associate  (AWS CSAA) exam. To be quite honest, I felt the biggest sense of relief, after walking out of the exam centre, not because I had passed, but because I had finally forced myself to sit this exam.

Since posting on LinkedIn about passing my AWS CSAA exam, I have received a lot of requests from people, asking for tips on how to prepare for this exam. In addition to replying to them, I thought best to also put up a blog post, to help others that might be preparing to sit for the same exam.

I have been a Microsoft person for years, so it was quite natural for me to transition to Microsoft Azure a few years back. Microsoft Azure, to put it simply, is awesome. It empowers its users to grow their IT much faster and beyond what their local datacenter’s can accomodate, with a simple, pay-as-you-use model. This model has helped so many businesses become successful, in such a short amount of time.

I start my New Year, each year, with a list of technologies that I had been introduced to, in the year that had just finished, but did not get a chance to properly get acquainted with. This list, then condenses into my list of todos for that year. Last year, AWS made it into my list and since then I have been spending time, finding out all about it.

Exams to me are not just a certification, but a chance to find more about a technology and to re-confirm my understanding of it. That is why I try my best to learn as much as possible about the technologies being tested in an exam. What better way to learn more about AWS then to do the AWS Certified Solutions Architect – Associate exam 😉

I got my training material from https://acloud.guru Ryan Kroonenburg teaches an awesome course to get one prepared to sit for the exam. The course is not too expensive either, well within the budget of anyone.

However, a note of caution. If you are very new to AWS and think that just doing the course will be enough to pass the exam, then there is some news for you. The whole basis for the AWS exams is to test you on your hands on experience and just cramming the information and proceeding to sit for the exam will be preparing yourself for a fail. I would highly recommend doing the labs that Ryan takes you through in the course. AWS provides a free tier, which covers almost all that you will need to train for the AWS CSAA exam. You can make use of this to get lots of hands on training. Also, read the whitepapers and faqs. These provide detailed information about the AWS services. ACloud.guru also has forums where you can ask questions and also answer questions others have posted.

To recap, my tips for passing the AWS CSAA exam

  • purchase the ACloud.guru course for AWS CSAA
  • go through the videos in the ACloud.guru course at least twice and do all the labs
  • answer all the section quizzes, the Mega Quizzes and Final exam questions and do them till you get at least 90% correct
  • read the AWS whitepapers and FAQs
  • participate in the ACloud.guru forums

You will soon realise that you are confident enough to sit the exam and that is when you book it and go sit for it.

I would highly recommend not waiting too long to sit for your exam (I have learnt from my mistake). I find that if I wait for more than a month to book my exam, I normally start forgetting all that I have learnt, and have to go over the material all over again. Everyone is different, so this might not be true for you. I would suggest aiming for 1 month of training, however in the 2nd week, book your exam for a date 2 weeks away. Doing this will firstly put you in panic mode, but then you will soon realise that you don’t have much time to study, and will start studying more intensely. By the last week, if you still think you are not fully prepared, you have 72 hours to reschedule the exam. If you are going to reschedule, then don’t reschedule to a date too far in the future.

I wish you all the best for the exam.