Using Windows Azure as a QA environment

I've recently been tasked with investigating whether or not Microsoft's Windows Azure cloud service is suitable for a Quality Assurance (QA) environment for our company software.

TL;DR: Azure is currently unsuitable for our QA environment, because of Azure's requirement of running Sysprep.exe before creating an image. Sysprep, and the problems caused by it, make our Upgrade test workflow practically unachievable.

What we require from a QA environment:

• A Windows Server 2012 build server to kick off software builds and to store them. The build server needs to be up all the time.  
• Windows Server 2008 R2 images of Virtual Machines (VMs). Each software package we own needs two of these - an Install VM to test a fresh install, and an Upgrade VM to test an upgrade from the previous version.

Our QA process:

• At a scheduled time, the build server kicks off NAnt scripts which grabs our source code from our repository on the cloud, compiles it, and uses the WiX toolset to build an installer file for our software.
• The build server launches a VM without the software installed. This "Install" VM uses automatically launched NAnt scripts to grab the install file from the build server, install the software, and run NUnit tests. The Install VM writes back a result to the build server. The install VM is then reverted back to its previous state.
• If the Install VM's tests passed, the build server launches an "Upgrade" VM which has our software installed on it. The Upgrade VM uses the same install file to upgrade it's software, and tests are run. 
• If the Upgrade VM's tests passed, the build server saves the state of the Upgrade VM, and publishes the install file.

Windows Azure

Azure seems to have been developed as a one-model-fits-all cloud service. The bits of interest to our QA process are:

• Virtual machine instances: A virtual machine which can be running or stopped (shut down). You can only capture an image from an instance if the virtual machine is stopped. 
• Images: Captured images of virtual machine instances. Virtual machines can be launched from these images.
• Disks: Disks used by current or previous virtual machines. Disks have to be manually deleted after use.
• Cloud services: In the QA process, a cloud service is essentially just a DNS with a web address to access each Virtual Machine instance you launch.
• Storage: A storage account with a specific location. For QA purposes, we used only one storage account in one location. You pay for any data transfer between locations.
• Virtual networks: Used to put all of your QA VMs on the same network, so they can communicate.

Windows Azure's REST API

I only used Azure's REST API, which provides enough functionality to create cloud services, launch VMs from images, capture images from VMs and delete disks.

In a future post, I'll explain how to use the API to launch VMs and play around with them.

Problems with Azure

Azure's model was intuitive enough, but after fiddling around with their services for a few days, I realised the following problems.

• Azure provides no decent form of snapshotting. If you want to save the state of a VM, you need to run sysprep.exe, shut down the VM, capture an image, and launch it from scratch later on.
• Sysprep.exe is compulsory before creating an image from a VM. takes a very long time (15-20 minutes) to run. What's more, Sysprep.exe managed to irreversibly destroy an image. Some developers claim that sysprep has a "3 generation limit," - after you run it three times, you can't run it any more without error. This may have been what destroyed my image.
• Your main administrator account is reset when you launch a VM from an image. This means you lose your user's files, your settings, and so on. You need to create a secondary account with administrator privileges to get around this.
• In the sysprep.exe and image launching process, some registry items like the crucial autoadminlogon settings are lost. I attempted to get around this by writing a batch file which sets the autoadminlogon registry entries on startup, and then restarting the machine to apply them. It was during testing of this script that Sysprep destroyed my image, so I never got to see the result of my batch file.
• The REST API doesn't provide functionality to check the status of a VM. For instance, you can't check whether a VM is stopped or running, which you need to know to capture an image from a VM.  To get around this, your scripts to capture an image from a VM need to loop a preprogrammed amount of times, expect error codes, and wait until Azure returns a positive result code.
• There is currently no way of giving your VMs static IP addresses. You also can't use the REST API to check a VMs IP address on the virtual network, which would be a very useful feature, Microsoft. Cough.

Conclusion

With all of these problems, it is difficult, but possible, to host a QA environment on Azure, provided that the QA VMs don't need to save state after each test. If you'll be launching the same non-updated image every test, then Azure might work for you.

However, our upgrade test workflow was not achievable with Azure. With enough effort, it might have been, but Sysprep.exe and other Azure hurdles just made it too much effort.

In the next few posts I will get into some of the technical aspects of using Azure for QA environment.