April 7, 2018

Deploy a VSTS Build Server in AWS

How to build and deploy a Linux VSTS build server in AWS to automate AWS CLI tasks.

Introduction

Microsoft’s Visual Studio Team Services (VSTS) uses build agents to build and deploy software. VSTS offers two types of build agents, Microsoft hosted agents and self-hosted agents. The advantages of hosted agents is that Microsoft takes cares of the maintenance and upgrades. However, sometimes the limitations of the hosted agents makes the self-hosted agent a better option.

In this post we demonstrate how to build and deploy a self-hosted agent running on Ubuntu on an AWS nano-instance. The build server is configured with the AWS CLI tools to enable the automation of AWS services from a VSTS CI/CD pipeline.

Prerequisites

This post assumes you have accounts setup for the following services:

Generate a Personal Access Token

The use of tools such as Git that do not use Microsoft accounts or Azure AD authentication require a personal access tokens (PAT) to be manually created in VSTS. Personal access tokens are considered more secure than alternative credentials such as username and password.

From the VSTS Console

  1. Logon to the VSTS console and from your profile navigate to the Security settings
  2. Click on Personal access tokens
  3. Click on Add
  4. Enter an appropriate description. Eg aws_vsts_build_server
  5. Click on Selected scopes and ensure all scopes are cleared
  6. Select Agent Pools (read, manage)
  7. If the agent will be a deployment group agent, also select Deployment group (read, manage)
  8. Click Create Token
  9. Take note of the token, it will be used later to configure the agent

Launch an AWS Linux Instance

We will use the AWS console to launch a Ubuntu Amazon Machine Image (AMI). For AWS orchestration tasks such as building and deploying websites to S3 and CloudFront, a nano or micro instance type is acceptable.

From the AWS Console

  1. Navigate to Services and select EC2
  2. Click on Launch Instance
  3. Find the Ubuntu Server 16.04 LTS (HVM), SSD … Amazon Machine Image then click Select
  4. Select the t2.nano (or t2.micro if eligible for the free usage tier) instance type then click on Next: Configure Instance Details
  5. Select an existing IAM role or click on Create new IAM role then click on Next: Add Storage
  6. Keep the defaults then click on Next: Add Tags
  7. Keep the defaults then click on Next: Configure Security Group
  8. The SSH rule has been added automatically, under Source change Custom to MyIP then click on Review and Launch
  9. Click Launch and when prompted select Create new key pair in the drop down and enter an appropriate name. Eg. aws_vsts_build_server
  10. Click on Download Key Pair and save in your home directory
  11. Click on Launch Instances
  12. Click on View Instances
  13. Wait for the instance state to change to running, select the instance, and take note of the IPv4 Public IP

SSH to AWS Instance

To configure our new AWS instance we need to establish an SSH session from our desktop. For Mac and Linux users this is straight forward. In the past Windows users had to install a third party SSH client like Putty, however, with the recent Windows 10 updates, Microsoft has included OpenSSH as a standard feature.

From a Windows 10 Desktop

If you do not already have an SSH client installed use the built in OpenSSH client:

  1. Open Apps & features and click on Manage optional features
  2. Select OpenSSH Client
  3. Click Install

Before we can use the AWS key pair we downloaded previously in OpenSSH we need to lock down the permissions on the PEM file:

  1. Right click on aws_vsts_build_server.pem file downloaded in the previous step and select Properties
  2. Select Security tab
  3. Click on Advanced
  4. Click on Disable inheritance
  5. Select the first option Convert inherited permissions into explicit permissions on this object
  6. Remove all permission entries other than the owner, then click OK to close the Advanced settings, the click OK to close the file Properties

We are now ready to establish an SSH session to our AWS instance. Open a new command prompt, navigate to the downloaded the aws_vsts_build_server.pem file and establish an SSH session to the AWS instance using the IPv4 Public IP you noted down from the AWS console:

ssh -i "aws_vsts_build_server.pem" ubuntu@IPV4_PUBLIC_IP

Type yes when prompted to continue connecting.

From an OSX or Linux Client

Open a new terminal and establish an SSH session to our new AWS instance:

cd ~
chmod 400 aws_vsts_build_server.pem
ssh -i "aws_vsts_build_server.pem" ec2-user@IPV4_PUBLIC_IP

Type yes when prompted to continue connecting.

Install and Configure Build Server

It is now time to configure the build server, first we install the AWS CLI and any dependencies. We then download and configure the VSTS agent.

Install and Configure AWS CLI

From the SSH session download and install the AWS CLI:

curl -O https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
sudo pip install awscli --upgrade

Note we do not need to configure credentials as EC2 has assumed a role for our instance. We do however need to set the default region in the aws-cli config file:

aws configure

Enter your default region name when prompted Eg: ap-southeast-2. Accept the defaults i.e. [None], for other questions.

Install and Configure VSTS Agent

From the ssh session we install the VSTS agent dependencies:

sudo apt-get install -y libunwind8 libcurl3
sudo apt-add-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

And then install the VSTS agent:

sudo wget https://vstsagentpackage.azureedge.net/agent/2.136.0/vsts-agent-linux-x64-2.136.0.tar.gz
mkdir myagent && cd myagent
tar xzf ../vsts-agent-linux-x64-2.136.0.tar.gz
./config.sh
  1. Enter the server URL when prompted in the form of https://youraccount.visualstudio.com
  2. Accept the default [PAT] for authentication type
  3. Enter the Personal Access Token you noted above in the VSTS console
  4. Accept the default [default] for the agent pool
  5. Enter a descriptive name when prompted for the agent name such as: aws_vsts_build_server
  6. Accept the default [_work] for the work folder

Next we configure the agent as a systemd service. Ensure you are in the ~/myagent directory and run the following:

sudo ./svc.sh install
sudo ./svc.sh start

You can now exit the SSH session.

Verify VSTS Agent Availability

At this stage the VSTS build server should be available with a status of online in the VSTS project settings console. Eg: https://youraccount.visualstudio.com/yourproject/_settings/agentqueues

References

The following resources were referenced when creating this post: