A scenario-based tutorial for Azure Kubernetes Service – Part 2

Introduction

In this blog, we will dig a little deeper into Azure Kubernetes Service (AKS). What better way to do this than by building an AKS cluster ourselves! Just a heads-up, I will be using terminology that was introduced in part 1 of this mini-blog series. If you haven’t read it, or need a refresher, you can access it at https://nivleshc.wordpress.com/2019/03/04/a-scenario-based-tutorial-for-azure-kubernetes-service-part-1

Let’s start by describing the AKS cluster architecture. The diagram below provides a great overview.

(Image copied from https://docs.microsoft.com/en-au/azure/aks/media/concepts-clusters-workloads/cluster-master-and-nodes.png)

The AKS Cluster is made up of two components. These are described below

  • cluster master node is an Azure managed service, which takes care of the Kubernetes service and ensures all the application workloads are properly running.
  • node is where the application workloads run.

The cluster master node is comprised of the following components

  • kube-apiserver – this api server provides a way to interface with the underlying Kubernetes API. Management tools such as kubectl or Kubernetes dashboard interact with this to manage the Kubernetes cluster.
  • etcd – this provides a key value store within Kubernetes, and is used for maintaining state of the Kubernetes cluster and state
  • kube-scheduler – the role of this component is to decide which nodes the newly created or scaled up application workloads can run on, and then it starts these workloads on them.
  • kube-controller-manager – the controller manager looks after several smaller controllers that perform actions such as replicating pods and handing node operations

The node is comprised of the following

  • kubelet – this is an agent that handles the orchestration requests from the cluster master node and also takes care of scheduling the running of the requested containers
  • kube-proxy – this component provides networking services on each node. It takes care of routing network traffic and managing IP addresses for services and pods
  • container runtime – this allows the container application workloads to run and interact with other resources within the node.

For more information about the above, please refer to https://docs.microsoft.com/en-au/azure/aks/concepts-clusters-workloads

Now that you have a good understanding of the Kubernetes architecture, lets move on to the preparation stage, after which we will deploy our AKS cluster.

Preparation

AKS subnet size

AKS uses a subnet to host nodes, pods, and any other Kubernetes and Azure resources that are created for the AKS cluster. As such, it is extremely important that the subnet is appropriately sized, to ensure it can accommodate the resources that will be initially created, and still have enough room for any future updates.

There are two networking methods available when deploying an Azure Kubernetes Service cluster

  • Kubenet
  • Azure Container Networking Interface (CNI)

AKS uses kubnet by default, and in doing so, it automatically creates a virtual network and subnets that are required to host the pods in. This is a great solution if you are learning about AKS, however if you need more control, it is better to go with Azure CNI. With Azure CNI, you get the option to use an existing virtual network and subnet or you can create a custom one. This is a much better option, especially when deploying into a production environment.

In this blog, we will use Azure CNI.

The formula below provides a good estimate on how large your subnet must be, in order to accommodate your AKS resources.

Subnet size = (number of nodes + 1) + ((number of nodes + 1) * maximum number of pods per node that you configure)

When using Azure CNI, by default each node is setup to run 30 pods. If you need to change this limit, you will have to deploy your AKS cluster using Azure CLI or Azure Resource Manager templates.

Just as an example, for a default AKS cluster deployment, using Azure CNI with 4 nodes, the subnet size at a minimum must be

IPs required = (4 + 1) + ((4+ 1) * (30 pods per node)) = 5 + (5 * 30) = 155

This means that the subnet must be at least a /24.

For this blog, create a new resource group called myAKS-resourcegroup. Within this new resource group, create a virtual network called AKSVNet with an address space of 10.1.0.0/16. Inside this virtual network, create a subnet called AKSSubnet1 with an address range of 10.1.3.0/24.

Deploying an Azure Kubernetes Service Cluster

Let’s proceed on to deploying our AKS cluster.

  1. Login to your Azure Portal and add Kubernetes Service
  2. Once you click on Create, you will be presented with a screen to enter your cluster’s configuration information
  3. Under Basics
  • Choose the subscription into which you want to deploy the AKS cluster
  • Choose the resource group into which you want to deploy the AKS cluster. One thing to point out here is that the cluster master node will be deployed in this resource group, however a new resource group with a name matching the naming format MC_<AKS master node resource group name>_<AKS cluster name>_region will be created to host the nodes where the containers will run (if you use the values specified in this blog, your node resource group will be named MC_myAKS-resourcegroup_mydemoAKS01_australiaeast)
  • Provide the Kubernetes cluster name (for this blog, let’s call this mydemoAKS01)
  • Choose the region you want to deploy the AKS cluster in (for this blog, we are deploying in australiaeast region)
  • Choose the Kubernetes version you want to deploy (you can choose the latest version, unless there is a reason to choose a specific version)
  • DNS name prefix – for simplicity, you can set this to the same as the cluster name
  • Choose the Node size. (for this blog, lets choose D2s v3 (2 vcpu, 8 GB memory)
  • Set the Node count to 1 (the Node count specifies the number of nodes that will be initially created for the AKS cluster)
  • Leave the virtual nodes to disabled

Under Authentication

  • Leave the default option to create a service principal (you can also provide an existing service principal, however for this blog, we will let the provisioning process create a new one for us)
  • RBAC allows you to control who can view the Kubernetes configuration (kubeconfig) information and to limit the permissions that they have. For now, leave RBAC turned off

Under Networking

  • Leave HTTP application routing set to No
  • As previously mentioned, by default AKS uses kubenet for networking. However, we will use Azure CNI. Change the Network configuration from Basic to Advanced
  • Choose the virtual network and subnet that was created as per the prerequisites (AKSVNet and AKSSubnet1)
  • Kubernetes uses a separate address range to allocate IP addresses to internal services within the cluster. This is referred to as Kubernetes service address range. This range must NOT be within the virtual network range and must not be used anywhere else. For our purposes we will use the range 10.2.4.0/24. Technically, it is possible to use IP addresses for the Kubernetes service address range from within the cluster virtual network, however this is not recommended due to potential of IP address overlaps which could potentially cause unpredictable behaviour. To read more about this, you can refer to https://docs.microsoft.com/en-au/azure/aks/configure-azure-cni.
  • Leave the Kubernetes DNS service IP address as the default 10.2.4.10 (the default is set to the tenth IP address within the Kubernetes service address range)
  • Leave the Docker bridge address as the default 172.17.0.1/16. The Docker Bridge lets AKS nodes communicate with the underlying management platform. This IP address must not be within the virtual network IP address range of your cluster, and shouldn’t overlap with other address ranges in use on your network

Under Monitoring

  • Leave enable container monitoring set to Yes
  • Provide an existing Log Analytics workspace or create a new one

Under Tags

  • Create any tags that need to be attached to this AKS cluster
     4.  Click on Next: Review + create to get the settings validated.
After validation has successfully passed, click on Create.
Just be aware that it can take anywhere from 10 – 15 minutes to complete the AKS cluster provisioning.

While you are waiting

During the AKS cluster provisioning process, there are a number of things that are happening under the hood. I managed to track down some of them and have listed them below.

  • Within the resource group that you specified for the AKS cluster to be deployed in, you will now see a new AKS cluster with the name mydemoAKS01
  • If you open the virtual network that the AKS cluster has been configured to use and click on Connected devices, you will notice that a lot of IP addresses that have been already allocated.

    I have noticed that the number of IP addresses equals

    ((number of pods per node) + 1) * number of nodes

   FYI – for the AKS cluster that is being deployed in this blog, it is 31

  • A new resource group with the name complying to the naming format MC_<AKS master node resource group name>_<AKS cluster name>_region will be created. In our case it will be called MC_myAKS-resourcegroup_mydemoAKS01_australiaeast. This resource group will contain the virtual machine for the node (not the cluster master node), including all the resources that are needed for the virtual machines (availability set, disk, network card, network security group)

What will this cost me?

The cluster master node is a managed service and you are not charged for it. You only pay for the nodes on which the application workloads are run (these are those resources inside the new resource group that gets automatically created when you provision the AKS cluster).

In the next blog, we will delve deeper into the newly deployed AKS cluster, exposing its configuration using command line tools.

Happy sailing and till the next time, enjoy!

Advertisements

Using Ansible to deploy an AWS environment

Background

Over the past few weeks, I have been looking at various automation tools for AWS. One tool that seems to get a lot of limelight is Ansible, an open source automation tool from Red Hat. I decided to give it a go, and to my amazement, I was surprised at how easy it was to learn Ansible, and how powerful it can be.

All that one must do is to write up a list of tasks using YAML notation in a file (called a playbook) and get Ansible to execute it. Ansible reads the playbook and executes the tasks in the order that they are written. Here is the biggest advantage, there are no agents to be installed on the managed computers! Ansible connects to each of the managed computers using ssh or winrm.

Another nice feature of Ansible is that it supports third party modules. This allows Ansible to be extended to support many of the services that it natively does not understand.

In this blog, we will be focusing on one of the third-party modules, the AWS module. Using this, we will use Ansible to deploy an environment within AWS.

Scenario

For this blog, we will use Ansible to provision an AWS Virtual Private Cloud (VPC) in the North Virginia (us-east-1) region. Within this VPC, we will create a public and a private subnet. We will then deploy a jumphost in the public subnet and a server within the private subnet.

Below is a diagram depicting what will be done.

Figure 1: Environment that will be deployed within AWS using Ansible Playbook

Preparation

The computer that is used to run Ansible to manage all other computers is referred to as the control machine. Currently, Ansible can be run from any machine with Python 2 (version 2.7) or Python 3 (version 3.5 or higher) installed. The Ansible control machine can run the following operating systems

  • Red Hat
  • Debian
  • CentOS
  • macOS
  • any of the BSD variants

Note: Currently windows operating system is not supported for running the control machine.

For this blog, I am using a MacBook to act as the control machine.

Before we run Ansible, we need to get a few things done. Let’s go through them now.

  1. We will use pip (Python package manager) to install Ansible. If you do not already have pip installed, run the following command to install it
    sudo easy_install pip
  2. With pip installed, use the following command to install Ansible
    sudo pip install ansible

    For those that are not using macOS for their control machine, you can get the relevant installation commands from https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html.

  3. Next, we must install the AWS Command Line Interface (CLI) tools. Use the following command for this.
    sudo pip install awscli

    More information about the AWS CLI tools is available at https://aws.amazon.com/cli/

  4. To provision items within AWS, we need to provide Ansible with a user account that has the necessary permissions. Using the AWS console, create a user account ensuring it is assigned an access key and a secret access key. At a minimum, this account must have the following policies assigned to it.
    AmazonEC2FullAccess
    AmazonVPCFullAccess

    Note: As this is a privileged user account, please ensure that the access key and secret access key is kept in a safe place.

  5. To provision AWS Elastic Compute Cloud (EC2) instances, we require key pairs created in the region that the EC2 instances will be deployed in. Ensure that you already have key pairs for the North Virginia (us-east-1) region. If not, please create them.

Instructions

Create an Ansible Playbook

Use the following steps to create an Ansible playbook to provision an AWS environment.

Open your favourite YAML editor and paste the following code

The above code instructs Ansible that it should connect to the local computer, to run all the defined tasks. This means that Ansible modules will use the local computer to connect to AWS APIs in order to carry out the tasks.

Another thing to note is that we are declaring two variables. These will be used later in the playbook.

  • vpc_region – this is the AWS region where the AWS environment will be provisioned (currently set to us-east-1)
  • my_useast1_key – provide the name of your key pair for the us-east-1 region that will be used to provision EC2 instances

Next, we will define the tasks that Ansible must carry out. The format of the tasks is as follows

  • name – this gives a descriptive name for the task
  • module name – this is the module that Ansible will use to carry out the task
  • module Parameters – these are parameters passed to the module, to carry out the specific task
  • register – this is an optional keyword and is used to record the output that is returned from the module, after the task has been carried out.

Copy the following lines of code into your YAMl file.

The above code contains two tasks.

  • the first task creates an AWS Virtual Private Cloud (VPC) using the ec2_vpc_net module. The output of this module is recorded in the variable ansibleVPC using the register command
  • the second task outputs the contents of the variable ansibleVPC using the debug command (this displays the output of the previous task)

Side Note

  • Name of the VPC has been set to ansibleVPC
  • The CIDR block for the VPC has been set to 172.32.0.0/16
  • The state keyword controls what must be done to the VPC. In our case, we want it created and to exist, as such, the value for state has been set to present.
  • The region is being set by referencing the variable that was defined earlier. Variables are referenced with the notation “{{ variable name }}”

Copy the following code to create an AWS internet gateway and associate it with the newly created VPC. The second task in the below code displays the result of the internet gateway creation.

The next step is to create the public and private subnets. However, instead of hardcoding the availability zones into which these subnets will be deployed, we will pick the first availability zone in the region for our public and the second availability zone in the region for our private subnet. Copy the following code into your YAML file to show all the availability zones that are present in the region, and which ones will be used for the public and private subnets.

Copy the following code to create the public subnet in the first availability zone in us-east-1 region. Do note that we are provisioning our public subnet with CIDR range 172.32.1.0/24

Copy the following code to deploy the private subnet in the second availability zone in us-east-1 region. It will use the CIDR range 172.32.2.0/24

Hold on! To make a public subnet, it is not enough to just create a subnet. We need to create routes from that subnet to the internet gateway! The below code will address this. The private subnet does not need any such routes, it will use the default route table.

As planned, we will be deploying jumphosts within the public subnet. By default, you won’t be able to externally connect to the EC2 instances deployed within the public subnet because the default security group does not allow this.

To remediate this, we will create a new security group that will allow RDP access and assign it to the jumphost server. For simplicity, the security group will allow RDP access from anywhere, however please ensure that for your environment, you have locked it down to a few external IP addresses.

Phew! Finally, we are ready to deploy our jumphost! Copy the following code for this

I would like to point out a few things

  • The jumphost is running on a t2.micro instance. This instance type is usually sufficient for a jumphost in a lab environment, however if you need more performance, this can be changed (changing the instance type from t2.micro can take you over the AWS free tier limits and subsequently add to your monthly costs)
  • The image parameter refers to the AMI ID of the Windows 2016 base image that is currently available within the AWS console. AWS, from time to time, changes the images that are available. Please check within the AWS console to ensure that the AMI ID is valid before running the playbook
  • Instance tags are tags that are attached to the instance. In this case, the instance tags have been used to name the jumphost win2016jh.

Important Information

The following parameters are extremely important, if you do not intend on deploying a new EC2 instance for the same server every time you re-run this Ansible playbook.

exact_count – this parameter specifies the number of EC2 instances of a server that should be running whenever the Ansible playbook is run. If the current number of instances doesn’t match this number, Ansible either creates new EC2 instances for this server or terminates the extra EC2 instances. The servers are identified using the count_tag

count_tag – this is the instance tag that is used to identify a server. Multiple instances of the same server will have the same tag applied to them. This allows Ansible to easily count how many instances of a server are currently running.

Next, we will deploy the servers within the private subnet. Wait a minute! By default, the servers within the private subnet will be assigned the default security group. The default security group allows unrestricted access to all EC2 instances that have been attached to the default security group. However, since the jumphost is not part of this security group, it will not be able to connect to the servers in the private subnet!

Let’s remediate this issue by creating a new security group that will allow RDP access from the public subnet to the servers within the private subnet (in a real environment, this should be restricted further, so that the incoming connections are from particular servers within the public subnet, and not from the whole subnet itself). This new security group will be associated with the servers within the private subnet.

Copy the following code into your YAML file.

We are now at the end of the YAML file. Copy the code below to provision the windows 2016 server within the private subnet (the server will be tagged with name=win2016svr)

Save the playbook with a meaningful name. I named my playbook Ansible-create-AWS-environment.yml

The full Ansible playbook can be downloaded from https://gist.github.com/nivleshc/344dca91e3d0349c8a359b03853886be

Running the Ansible Playbook

Before we run the playbook, we need to tell Ansible about all the computers that are within the management scope. This is done using an inventory file, which contains a group name within square brackets eg [webservers] and below that, all the computers that will be in that group. Then in the playbook, we just target the group, which in turn targets all the computers in that group.

However, in our scenario, we are directly targeting the local computer (refer to the second line in the YAML file that shows hosts: localhost). In this regard, we can get away with not providing an inventory file. However, do note that doing so will mean that we can’t use anything other than localhost to reference a computer within our playbook.

Let’s create an inventory file called hosts in the same folder as where the playbook is saved. The contents of the file will be as listed below.

[local]
localhost

We are ready to run the playbook now.

Open a terminal session and change to the folder where the playbook was saved.

We need to create some environment variables to store the user details that Ansible will use to connect to AWS. This is where the access key and secret access key that we created initially will be used. Run the following command

export AWS_ACCESS_KEY_ID={access key id}
export AWS_SECRET_ACCESS_KEY={secret access key}

Now run the playbook using the following command (as previously mentioned, we could get away with not specifying the inventory file, however this means that we only can use localhost within the playbook)

ansible-playbook -i hosts ansible-create-aws-environment.yml

You should now see each of the tasks being executed, with the output being shown (remember that after each task, we have a follow-up task that shows the output using the debug keyword? )

Once the playbook execution has completed, check your AWS console to confirm that the following items have been created within the us-east-1 (North Virginia) region

  • A VPC called ansibleVPC with the CIDR 172.32.0.0/16
  • An internet gateway called ansibleVPC_igw
  • A public subnet in the first availability zone with CIDR 172.32.1.0/24
  • A private subnet in the second availability zone with CIDR 172.32.2.0/24
  • A route table called rt_ansibleVPC_PublicSubnet
  • A security group for jumphosts called sg_ansibleVPC_publicsubnet_jumphost
  • A security group for the servers in private subnet called sg_ansibleVPC_privatesubnet_servers
  • An EC2 instance in the public subnet representing a jumphost named win2016jh
  • An EC2 instance in the private subnet representing a server named win2016svr

Once the provisioning is complete, to test, connect to the jumphost and then from there connect to the server within the private subnet.

Don’t forget to turn off the EC2 instances if you don’t intend on using them

Closing Remarks

Ansible is a great automation tool and can be used to both provision and manage infrastructure within AWS.

Having said that, I couldn’t find an easy way to do post provisioning tasks (eg assigning roles, installing additional packages etc) after the server has been provisioned, without getting Ansible to connect directly to the provisioned server. This can be a challenge if the Ansible control machine is external to AWS and the provisioned server is within an AWS private subnet. With AWS CloudFormation, this is easily done. If anyone has any advice on this, I would appreciate it if you can leave it in the comments below.

I will surely be using Ansible for most of my automations from now on.

Till the next time, enjoy!

Enabling Source Control for locally stored code using Git, Visual Studio Code and Sourcetree

Introduction

Coming from a system administration background, I am used to writing scripts to get mundane tasks done. Whenever I saw repeatable tasks, I saw an opportunity to script them, and pass them onto a junior to do 😉

However, writing scripts brings about its own challenges.

Ok, time to fess up 😉 Hands up those that have modified a script, only to realise that the modifications broke it! To make matters worse, you forgot to take a copy of the original!

Don’t worry, I have been in that boat, and can remember the countless hours I spent, getting the script back to what it was (mind you, I am not talking about a formal business change here, which is governed by strict change control, but about personal scripts, that you have created to make your daily tasks easier)

To make a copy of a script, I would normally suffix the file with the current time and date. This provided me with a timestamp of when I changed the file and a way of reverting my changes. However, there were instances when I was making backups of the modified script because I had tested a modification and it worked, however I didn’t want to risk breaking it when further modifying the file. Guess what, these are the times when I found I made the worst mistakes! I used to get so engrossed with my modifications that I would forget to make a backup of the changes and end up with an unworkable script. The only version to revert to was the original, which meant all my hard work went to waste!

This is why I started my search for a better change tracking system. One that will show me the changes I had made, and which will allow me to easily revert to a previous version.

Guess what! I think I just found this golden goose and it is truly amazing!

In this blog I will show you how you can use Git, an open source version control system,  to track changes to scripts stored locally on your computer. The main use of Git is for source control of files that a team contributes to. In these situations, a Git Server is used to store the repository.

Please ensure that the local folder you are tracking for source control is backed up either to the cloud or to an external hard disk.

For editing our code/script, we will use Microsoft’s Visual Studio Code, a free IDE that has Git support in-built. We will also use Sourcetree, Atlassian’s free Git client.

 

Introducing Git

Git is an awesome opensource distributed version control system. When working in a team, it allows you to have your files centrally managed, and at the same time, allowing multiple people to work on them. Team members can pull the repository to their local computer. They can also branch a part of the repository, update the files in that part and then merge them with the master. If there are no conflicts, Git will update the files in its repository. However, if there are conflicts, Git will inform that team member, showing them the conflicts. The team member then can either resolve the conflicts and then re-merge or discard their changes altogether.

If you want to read more on git, check out https://git-scm.com

To host the repositories for your team, two commonly used solutions are a Git Server or Visual Studio Teams Server. You can also use Github, however, your repositories will be public, unless you sign up for a paid account.

For personal use, you can store your git repositories in a local directory that is backed up to the cloud. For my personal projects, I use a Dropbox synchronised folder.

To use Git, you need to use a Git client. If you have a MacBook, a git client comes built-in. For windows, there are lots of clients available, however in my view, Sourcetree is one of the best (more about this abit later).

For MacBook users, below are some basic commands you can use from a terminal session

#change to directory where you will store your repository
cd /Users/tomj/Documents/git-repo/personalproject

#create a git repo in this folder
git init

#you can copy files into this folder
#to get git to start tracking the changes in the newly added files use the following command
git add .

 

As mentioned above, https://git-scm.com is an awesome site to learn more about Git.

You can also check out this page https://www.atlassian.com/git/tutorials/atlassian-git-cheatsheet for some quick git commands.

Using Sourcetree

For those that prefer a GUI client, I found that Sourcetree, from Atlassian, is an awesome Git client.

It gives you all the features of a good Git client plus it also shows you the history of all the changes made to the repository.

For this blog, I will be using Sourcetree to create and manage the Git repository.

So head over to https://www.sourcetreeapp.com to download and then install the client.

After installing Sourcetree, you will be prompted for a login account. Follow the links provided in the Sourcetree app to create a free Bitbucket account and then login.

Ok, lets begin.

Create a new repository

A repository is essentially a collection of files (or file) that we will track for changes. You can think of it as a directory.

To create a new repository, open Sourcetree.

From the menu, click on File and then click New. You will get the following screen.

Sourcetree_newRepo

Next, click on New and then click on Create Local Repository.

In the next window, for Destination Path, select the folder that will contain the scripts that you want to monitor for source control.

For  Name leave it to the default (name of the folder). Ensure the Type is Git and then click Create.

Guess what, thats all it takes to create a local repository! Simple ?

Once the repository is created, you will see a screen similar to the one shown below (my repository is called temp)

Sourcetree_newRepoCreated.

Double click on the newly created repository (as shown above). This will show the dashboard where everything happens 😉

Sourcetree_RepoDashboard

 

To see all the changes that have been made to the repository, click on History in the above screen.

Visual Studio Code

Ok, so we have created our repository and it is being monitored for changes. Now, we can start coding.

As mentioned above, we will be using Visual Studio Code, a free IDE from Microsoft. If you haven’t got it already, download it from https://code.visualstudio.com

Once installed, open Visual Studio Code.

From the menu, click on File and then click on  Open.  Next, choose the folder that you created the repository for above and then click on Open.

You will now see the folder structure, with all the files inside it in the left pane.

You can open any of the existing files or create new ones. For new ones, ensure you save them in the repository’s folder.

As soon as you save the file, you will notice the Source Control icon shows the number of changes that are currently ready to be staged (Source Control section is denoted by the “stethoscope” icon – ok it’s not really that but it surely looks like it 😉 )

VSC_SourceControl_Update

Now, one thing to note about Source Control via Git is that, you have to stage your changes. When you stage your changes, those changes will be written to the Git repository when you click Commit.

Click into the Source Control section and then under Changes click the + for each of the files, to stage the change.

VSC_SourceControl_Update_Changes

To commit the changes, enter a short description of what the changes were and then click on the tick at the top.

VSC_SourceControl_Update_Changes_Commit

That’s it. Your changes has now been successfully committed to the Git repository.

To view a history of all the changes that have been done to your repository, open Sourcetree and then click on History.

Notice the description column. This contains the comments you wrote when committing your staged changes. This provides a quick reminder of what the changes were. To drill down deeper into the changes, check the pane at the bottom right. Here, you will see the actual changes that were made (green denotes additions and red denotes deletion of characters). If there are multiple people committing to the same repo (as would be the case in a team), the names of each person will be shown beside each line in the History section.

Sourcetree_ViewHistory

Now, lets say that after you did your commit, you realised that you didn’t want that change, and in-fact you prefer what the file was before the commit. All you need to do is go into Sourcetree, in the History section, find the change and then right click on it and then click on Reverse commit. This reverses the commit and changes the file to what it previously was. If after that, you want to get back the change? Well, you can reverse the reverse commit 😉 (this is so much better than my method of copying the last suffixed version to the current version)

Soucetree_ReverseCommit

Closing Remarks

I am absolutely loving Git. It is an awesome tool and I would highly recommend it to each and everyone. For me personally, it helps in controlling the various changes I make to my code, with easy auditability and view of the changes I make between versions.

For teams, Git provides even more benefits. Using a central server (Git Server or Visual Studio Team Services) to host the Git repositories, the whole team can work on the files without blocking each other. The files will be stored centrally (actually with Git, when you pull a repo, you download the full repo to your local computer. If you merge your changes, the files are merged to the copy on the server). The changes to the files are easily trackable and there is an easy way to revert to a previous version should issues arise due to modifications.

I hope you embrace Git as I have and use it to track all your code changes.

Till the next time, Enjoy 😉

 

Ok Google Email me the status of all vms – Part 2

In my last blog, we configured the backend systems necessary for accomplishing the task of asking Google Home “OK Google Email me the status of all vms” and it sending us an email to that effect. If you haven’t finished doing that, please refer back to my last blog and get that done before continuing.

In this blog, we will configure Google Home.

Google Home uses Google Assistant to do all the smarts. You will be amazed at all the tasks that Google Home can do out of the box.

For our purposes, we will be using the platform IF This Then That or IFTTT for short. IFTTT is a very powerful platform as it lets you create actions based on triggers. This combination of triggers and actions is called a recipe.

Ok, lets dig in and create our IFTTT recipe to accomplish our task

1.1   Go to https://ifttt.com/ and create an account (if you don’t already have one)

1.2   Login to IFTTT and click on My Applets menu from the top

IFTTT_MyApplets_Menu

1.3   Next, click on New Applet (top right hand corner)

1.4   A new recipe template will be displayed. Click on the blue + this choose a service

IFTTT_Reicipe_Step1

1.5   Under Choose a Service type “Google Assistant”

IFTTT_ChooseService

1.6   In the results Google Assistant will be displayed. Click on it

1.7   If you haven’t already connected IFTTT with Google Assistant, you will be asked to do so. When prompted, login with the Google account that is associated with your Google Home and then approve IFTTT to access it.

IFTTT_ConnectGA

1.8   The next step is to choose a trigger. Click on Say a simple phrase

IFTTT_ChooseTrigger

1.9   Now we will put in the phrases that Google Home should trigger on.

IFTTT_CompleteTrigger

For

  • What do you want to say? enter “email me the status of all vms
  • What do you want the Assistant to say in response? enter “no worries, I will send you the email right away

All the other sections are optional, however you can fill them if you prefer to do so

Click Create trigger

1.10   You will be returned to the recipe editor. To choose the action service, click on + that

IFTTT_That

1.11  Under Choose action service, type webhooks. From the results, click on Webhooks

IFTTT_ActionService

1.12   Then for Choose action click on Make a web request

IFTTT_Action_Choose

1.13   Next the Complete action fields screen is shown.

For

  • URL – paste the webhook url of the runbook that you had copied in the previous blog
  • Method – change this to POST
  • Content Type – change this to application/json

IFTTT_CompleteActionFields

Click Create action

1.13   In the next screen, click Finish

IFTTT_Review

 

Woo hoo. Everything is now complete. Lets do some testing.

Go to your Google Home and say “email me the status of all vms”. Google Home should reply by saying “no worries. I will send you the email right away”.

I have noticed some delays in receiving the email, however the most I have had to wait for is 5 minutes. If this is unacceptable, in the runbook script, modify the Send-MailMessage command by adding the parameter -Priority High. This sends all emails with high priority, which should make things faster. Also, the runbook is currently running in Azure. Better performance might be achieved by using Hybrid Runbook Workers

To monitor the status of the automation jobs, or to access their logs, in the Azure Automation Account, click on Jobs in the left hand side menu. Clicking on any one of the jobs shown will provide more information about that particular job. This can be helpful during troubleshooting.

Automation_JobsLog

There you go. All done. I hope you enjoy this additional task you can now do with your Google Home.

If you don’t own a Google Home yet, you can do the above automation using Google Assistant as well.

Ok Google Email me the status of all vms – Part 1

Technology is evolving at a breathtaking pace. For instance, the phone in your pocket has more grunt than the desktop computers of 10 years ago!

One of the upcoming areas in Computing Science is Artificial Intelligence. What seemed science fiction in the days of Isaac Asimov, when he penned I, Robot seems closer to reality now.

Lately the market is popping up with virtual assistants from the likes of Apple, Amazon and Google. These are “bots” that use Artificial Intelligence to help us with our daily lives, from telling us about the weather, to reminding us about our shopping lists or letting us know when our next train will be arriving. I still remember my first virtual assistant Prody Parrot, which hardly did much when you compare it to Siri, Alexa or Google Assistant.

I decided to test drive one of these virtual assistants, and so purchased a Google Home. First impressions, it is an awesome device with a lot of good things going for it. If only it came with a rechargeable battery instead of a wall charger, it would have been even more awesome. Well maybe in the next version (Google here’s a tip for your next version 😉 )

Having played with Google Home for a bit, I decided to look at ways of integrating it with Azure, and I was pleasantly surprised.

In this two-part blog, I will show you how you can use Google Home to send an email with the status of all your Azure virtual machines. This functionality can be extended to stop or start all virtual machines, however I would caution against NOT doing this in your production environment, incase you turn off some machine that is running critical workloads.

In this first blog post, we will setup the backend systems to achieve the tasks and in the next blog post, we will connect it to Google Home.

The diagram below shows how we will achieve what we have set out to do.

Google Home Workflow

Below is a list of tasks that will happen

  1. Google Home will trigger when we say “Ok Google email me the status of all vms”
  2. As Google Home uses Google Assistant, it will pass the request to the IFTTT service
  3. IFTTT will then trigger the webhooks service to call a webhook url attached to an Azure Automation Runbook
  4. A job for the specified runbook will then be queued up in Azure Automation.
  5. The runbook job will then run, and obtain a status of all vms.
  6. The output will be emailed to the designated recipient

Ok, enough talking 😉 lets start cracking.

1. Create an Azure AD Service Principal Account

In order to run our Azure Automation runbook, we need to create a security object for it to run under. This security object provides permissions to access the azure resources. For our purposes, we will be using a service principal account.

Assuming you have already installed the Azure PowerShell module, run the following in a PowerShell session to login to Azure

Import-Module AzureRm
Login-AzureRmAccount

Next, to create an Azure AD Application, run the following command

$adApp = New-AzureRmADApplication -DisplayName "DisplayName" -HomePage "HomePage" -IdentifierUris "http://IdentifierUri" -Password "Password"

where

DisplayName is the display name for your AD Application eg “Google Home Automation”

HomePage is the home page for your application eg http://googlehome (or you can ignore the -HomePage parameter as it is optional)

IdentifierUri is the URI that identifies the application eg http://googleHomeAutomation

Password is the password you will give the service principal account

Now, lets create the service principle for the Azure AD Application

New-AzureRmADServicePrincipal -ApplicationId $adApp.ApplicationId

Next, we will give the service principal account read access to the Azure tenant. If you need something more restrictive, please find the appropriate role from https://docs.microsoft.com/en-gb/azure/active-directory/role-based-access-built-in-roles

New-AzureRmRoleAssignment -RoleDefinitionName Reader -ServicePrincipalName $adApp.ApplicationId

Great, the service principal account is now ready. The username for your service principal is actually the ApplicationId suffixed by your Azure AD domain name. To get the Application ID run the following by providing the identifierUri that was supplied when creating it above

Get-AzureRmADApplication -IdentifierUri {identifierUri}

Just to be pedantic, lets check to ensure we can login to Azure using the newly created service principal account and the password. To test, run the following commands (when prompted, supply the username for the service principal account and the password that was set when it was created above)

$cred = Get-Credential 
Login-AzureRmAccount -Credential $cred -ServicePrincipal -TenantId {TenantId}

where Tenantid is your Azure Tenant’s ID

If everything was setup properly, you should now be logged in using the service principal account.

2. Create an Azure Automation Account

Next, we need an Azure Automation account.

2.1   Login to the Azure Portal and then click New

AzureMarketPlace_New

2.2   Then type Automation and click search. From the results click the following.

AzureMarketPlace_ResultsAutomation

2.3   In the next screen, click Create

2.4   Next, fill in the appropriate details and click Create

AutomationAccount_Details

3. Create a SendGrid Account

Unfortunately Azure doesn’t provide relay servers that can be used by scripts to email out. Instead you have to either use EOP (Exchange Online Protection) servers or SendGrid to achieve this. SendGrid is an Email Delivery Service that Azure provides, and you need to create an account to use it. For our purposes, we will use the free tier, which allows the delivery of 2500 emails per month, which is plenty for us.

3.1   In the Azure Portal, click New

AzureMarketPlace_New

3.2   Then search for SendGrid in the marketplace and click on the following result. Next click Create

AzureMarketPlace_ResultsSendGrid

3.3   In the next screen, for the pricing tier, select the free tier and then fill in the required details and click Create.

SendGridAccount_Details

4. Configure the Automation Account

Inside the Automation Account, we will be creating a Runbook that will contain our PowerShell script that will do all the work. The script will be using the Service Principal and SendGrid accounts. To ensure we don’t expose their credentials inside the PowerShell script, we will store them in the Automation Account under Credentials, and then access them from inside our PowerShell script.

4.1   Go into the Automation Account that you had created.

4.2   Under Shared Resource click Credentials

AutomationAccount_Credentials

4.3    Click on Add a credential and then fill in the details for the Service Principal account. Then click Create

Credentials_Details

4.4   Repeat step 4.3 above to add the SendGrid account

4.5   Now that the Credentials have been stored, under Process Automation click Runbooks

Automation_Runbooks

Then click Add a runbook and in the next screen click Create a new runbook

4.6   Give the runbook an appropriate name. Change the Runbook Type to PowerShell. Click Create

Runbook_Details

4.7   Once the Runbook has been created, paste the following script inside it, click on Save and then click on Publish

Import-Module Azure
$cred = Get-AutomationPSCredential -Name 'Service Principal account'
$mailerCred = Get-AutomationPSCredential -Name 'SendGrid account'

Login-AzureRmAccount -Credential $cred -ServicePrincipal -TenantID {tenantId}

$outputFile = $env:TEMP+ "\AzureVmStatus.html"
$vmarray = @()

#Get a list of all vms 
Write-Output "Getting a list of all VMs"
$vms = Get-AzureRmVM
$total_vms = $vms.count
Write-Output "Done. VMs Found $total_vms"

$index = 0
# Add info about VM's to the array
foreach ($vm in $vms){ 
 $index++
 Write-Output "Processing VM $index/$total_vms"
 # Get VM Status
 $vmstatus = Get-AzurermVM -Name $vm.Name -ResourceGroupName $vm.ResourceGroupName -Status

# Add values to the array:
 $vmarray += New-Object PSObject -Property ([ordered]@{
 ResourceGroupName=$vm.ResourceGroupName
 Name=$vm.Name
 OSType=$vm.StorageProfile.OSDisk.OSType
 PowerState=(get-culture).TextInfo.ToTitleCase(($vmstatus.statuses)[1].code.split("/")[1])
 })
}
$vmarray | Sort-Object PowerState,OSType -Desc

Write-Output "Converting Output to HTML" 
$vmarray | Sort-Object PowerState,OSType -Desc | ConvertTo-Html | Out-File $outputFile
Write-Output "Converted"

$fromAddr = "senderEmailAddress"
$toAddr = "recipientEmailAddress"
$subject = "Azure VM Status as at " + (Get-Date).toString()
$smtpServer = "smtp.sendgrid.net"

Write-Output "Sending Email to $toAddr using server $smtpServer"
Send-MailMessage -Credential $mailerCred -From $fromAddr -To $toAddr -Subject $subject -Attachments $outputFile -SmtpServer $smtpServer -UseSsl
Write-Output "Email Sent"

where

  • ‘Service Principal Account’ and ‘SendGrid Account’ are the names of the credentials that were created in the Automation Account (include the ‘ ‘ around the name)
  • senderEmailAddress is the email address that the email will show it came from. Keep the domain of the email address same as your Azure domain
  • recipientEmailAddress is the email address of the recipient who will receive the list of vms

4.8   Next, we will create a Webhook. A webhook is a special URL that will allow us to execute the above script without logging into the Azure Portal. Treat the webhook URL like a password since whoever possesses the webhook can execute the runbook without needing to provide any credentials.

Open the runbook that was just created and from the top menu click on Webhook

Webhook_menu

4.9   In the next screen click Create new webhook

4.10  A security message will be displayed informing that once the webhook has been created, the URL will not be shown anywhere in the Azure Portal. IT IS EXTREMELY IMPORTANT THAT YOU COPY THE WEBHOOK URL BEFORE PRESSING THE OK BUTTON.

Enter a name for the webhook and when you want the webhook to expire. Copy the webhook URL and paste it somewhere safe. Then click OK.

Once the webhook has expired, you can’t use it to trigger the runbook, however before it expires, you can change the expiry date. For security reasons, it is recommended that you don’t keep the webhook alive for a long period of time.

Webhook_details

Thats it folks! The stage has been set and we have successfully configured the backend systems to handle our task. Give yourselves a big pat on the back.

Follow me to the next blog, where we will use the above with IFTTT, to bring it all together so that when we say “OK Google, email me the status of all vms”, an email is sent out to us with the status of all the vms 😉

I will see you in Part 2 of this blog. Ciao 😉

Script to shutdown servers

I run a lot of Microsoft virtual machines in Azure and also locally on my MacBook Pro. These are my lab machines, which I use for testing.

One of the issues with having many virtual machines is orderly shutting them down. It can be a pain to go through each of them and shutting them down.

To circumvent this, I wrote a small PowerShell script, which does it all for me 🙂

The script requires the following

$serverlist contains the hostnames of the servers that you want to shutdown (in the order they need to be shutdown)

$server_domainname this is the domain name that the servers are part of.

servername and and server_domainname is used to figure out the server fqdn, which is then used to shutdown that server.

Run the script from a computer that can connect to the servers. Ensure you are logged on as an account that has permissions to shutdown the servers.

The script will go through the list of servers contained in $serverlist and check if they are online. If they are online, then it will try to shut them down.

Do note that these servers will be forced to shutdown, so anything open on those servers will be lost, if not saved.

Once all the online servers have been shutdown, you will be asked if you want to shutdown the computer you are running the script from. You can press Enter to continue or CTRL+C to skip shutting down the computer you are logged on.

Hope this script comes in handy to others

Re-execute the UserData script in an AWS Windows Instance

Bootstrapping is an awesome way of customising your instances in AWS (similar capability exists in Azure).

To enable bootstrapping, while configuring the launch instance, in Step 3: Configure Instance Details scroll down to the bottom and then expand Advanced Details.

You will notice a User data text box. This is where you can provide your bootstrap script. The script will be run when your instance is first launched.

AWS_BootstrapScript

I went ahead and entered my script in the text box and proceeded to complete my instance configuration. Once my instance was running, I initiated a Remote Desktop connection to it, to confirm that my script had run. Unfortunately, I couldn’t see any customisations (which meant my script didn’t run)

Thinking that the instance had not been able to access the user data, I opened up Internet Explorer and then browsed to the following url (this is an internal url that can be used to access the user-data)

http://169.254.169.254/latest/user-data/

I was able to successfully access the user-data, which meant that there were no issues with that.  However when checking the content, I noticed a typo! Aha, that was the reason why my customisations didn’t happen.

Unfortunately, according to AWS, user-data is only executed during launch (for those that would like to read, here is the official AWS documentation). To get the fixed bootstrap script to run, I would have to terminate my instance and launch a new one with the corrected script (I tried re-booting my windows instance after correcting my typo, however it didn’t run).

I wasn’t very happy on terminating my current instance and then launching a new one, since for those that might not be aware, AWS EC2 compute charges are rounded up to the next hour. Which means that if I terminated my current instance and launched a new one, I would be charged for 2 x 1hour sessions instead of just 1 x 1 hour!

So I set about trying to find another solution. And guess what, I did find it 🙂

Reading through the volumes of documentation on AWS, I found that when Windows Instances are provisioned, the service that does the customisations using user-data is called EC2Config. This service runs the initial startup tasks when the instance is first started and then disables them. HOWEVER, there is a way to re-enable the startup tasks later on 🙂 Here is the document that gives more information on EC2Config.

The Amazon Windows AMIs include a utility called EC2ConfigService Settings. This allows you to configure EC2Config to execute the user-data on next service startup. The utility can be found under All Programs (or you can search for it).

AWS_EC2ConfigSettings_AllApps

AWS_EC2ConfigSettings_Search

Once Open, under General you will see the following option

Enable UserData execution for next service start (automatically enabled at Sysprep) eg. or <powershell></powershell>

AWS_EC2ConfigSettings

Tick this option and then press OK. Then restart your Windows Instance.

After your Windows Instance restarts, EC2Config will execute the userData (bootstrap script) and then it will automatically remove the tick from the above option so that the userData is not executed on subsequent restarts (or service starts)

There you go. A simple way to re-run your bootstrap scripts on an AWS Windows Instance without having to terminate the current instance and launching a new one.

There are other options available in the EC2ConfigService Settings that you can explore as well 🙂