Building the build environment
Like many software companies we have evolved to use a number of technologies when developing our SaaS solutions, PHP, NodeJS, AngularJS, Ruby, and Java, to name a few. We like to practise continuous integration here at Talis, therefore we run build plans for all of our applications ensuring that artifacts are built, tested, and are ready for deployment.
We currently use Atlassian Elastic Bamboo but this could easily apply to other CI solutions such as Jenkins. Initially our build plans consisted of a bunch of AMIs sourced from various places, running various versions of packages obtained from various software repositories.. you get the idea. It is quite easy for configuration to drift amongst build projects, never mind the potential configuration drift between build servers, staging, uat, preview, and production environments. To counteract this, we really wanted to use the same well practised provisioning in our production environments all the way back down the chain to the build servers, even for the stack to be available to build locally on a developer machine.
This post is about creating dynamic build environments for Elastic Bamboo using Puppet profiles that are in-line with current production environments.
There are some major benefits to building your build environment dynamically on demand.
- Rebuilding EBS backed or Instance Store AMIs when adding/changing packages can be a time consuming process, never mind documentative.
- Building software using the same provisioning and configuration management as production, tests the integrity of your build processes.
- Configuration changes at the build level can propagate consistently up the chain keeping environments synced.
Currently we are using Elastic Bamboo for building applications. A single build AMI is shared amongst all build plans, unfortunately though there does not seem to be a way to tell Bamboo to link a plan to a running instance, therefore using the same AMI image configuration amongst build plans will result in cross contamination. So we still have to have a specific image configuration for each build project, even though the image configuration uses the same AMI, each image configuration has a custom variable to keep them unique to a build project.
The key here is that the mechanics on how the instance is provisioned is in-line with how we provision in development, testing, and production environments. The instance starts up by pulling and running a boot script from S3, the script installs a version of Puppet and sets the Puppet config required. The script also sets a fact in Facter that is available to Puppet role => bamboo-build such as:
sudo su -c "echo "role=bamboo-build" > /etc/facter/facts.d/role.txt"This could be done in the Instance Start Up Script found in Bamboo Admin / Image Configurations.
In the Puppet manifests directory there is a default.pp to catch any node that is not classified by the ENC. Here the Elastic Bamboo agent is installed on any instance that calls in with the role set as bamboo-build. A module to install the agent can be found here
node "default" {
case $::role {
'bamboo-build' : { class { 'profile::bamboo-build': } }
'projecta-dev' : { class { 'profile::projecta-dev': } }
'projectb-dev' : { class { 'profile::projectb-dev': } }
}
}When Bamboo launches the instance the role fact is set as bamboo-build, this tells Puppet to install the Elastic Bamboo agent. Once the agent is installed and running, the instance is now ready for Bamboo to execute the build plan.
So now we have our Bamboo agent without any of the software required to run our build plan, therefore the first step in the build plan is to change the role fact and install the required dependencies for our build. The first task in our build plan is a script task. This sets the role fact to the required value for the build plan, then runs the Puppet agent to install the build dependencies, and finally sets the return code to 0 if the Puppet agent returns a return code of 2, otherwise Bamboo would fail the build.
After Puppet installs class profile::projecta-dev, our plan is now ready to grab the source code and build! Obviously there is a cost of slightly longer build times as the instances are built on demand instead of launching an AMI that already has the required build stack. This same process could be used to create a new instance store or EBS backed AMI that is then used for further build projects, but we think the slight increase in build times is worth the continual testing of our full stack.
This keeps all of our build environments current, and also enables engineers to build environments locally using Vagrant or similar by specifying the build role fact and running Puppet.