On Configuring Automated Builds on an Amazon EC2 micro-instance using Jenkins, SVN and Apache

I am a big fan of the continuous integration approach with using tools and build scripts to automate the build process for programs. Combine this with comprehensive unit testing and static analysis provides some easy ways to improve the quality of code.

So I wanted to implement a continuous integration set up, partly as a way to improve my infrastructure around the PunchClock project, as well as a chance to explore some new technologies, especially Amazon Web Services.

What I settled on was to create a "build machine" using Jenkins CI, formerly Hudson, an open source program specifically for automating building software in a continuous integration environment. On the same server I would also host my Subversion repository (SVN), and being that it is an Amazon EC2 instance I have less worry of a possibility of a catastrophic loss of data from my current Ubuntu box at home. Both SVN and the Jenkins web application would sit behind an Apache2 Web Server. All of this would be on a Debian Linux distro running as an EC2 instance. (I chose to go with Debian purely to gain exposure to other distros besides Ubuntu). I learned a lot from this set up, the AWS/EC2 was actually more challenging than you would normally find from a hosted server, but their AWS Management Console allows you to do most operations pretty easily.


Prereqs

1. Install Apache2 if not present
   On the default Debian install Apache is present, but on the Debian EC2 AMI I used it wasn't installed.
   apt-get install apache2
   
2. Install Java 1.6 (update 25)
   You'll need to edit your /etc/apt/sources.list see http://jayraj87.blogspot.com/2010/02/installing-sun-java6-jdk-in-debian.html
   
3. Install Ant 1.8.0
   
   • installs version 1.8.0 to /usr/bin/ant (/usr/share/ant)
     apt-get install ant
     
   • copy junit.jar to ANT_HOME\lib (junit 3.8.2 or later download here)
4. Install SVN and Repository
   
   • I installed SVN and then moved my existing repository to a location (this is a good article on moving svn: http://linux.justinhartman.com/Moving_a_Subversion_repository_to_another_server) . 
   • Also configured Apache2 with dav_svn for webaccess and basic authentication (http://davidwinter.me/articles/2006/02/16/subversion-over-apache-2-on-ubuntu/)
5. Before you begin I assume you have your own working ANT script that would compile, run JUnit tests, run any static analysis utilities (e.g. FindBugs or CheckStyle) and package your project into artifacts for distribution. The PunchClock application had an existing ANT build.xml that I started from.
   

Installation

All of these steps I honed on a default Debian install. I tested this on VMWare Player running the 6.0.1a "Squeeze" Debian install, from an iso file found on BitTorrent. Once I had got the steps down, basically the same applied to installing on the Debian instance on Amazon EC2 except for a few issues. All of these steps were performed as root unless otherwise noted.

1. Install Jenkins LTS
   
   • 
     wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
     
     • Add site to debian packages /etc/apt/sources.list: deb http://pkg.jenkins-ci.org/debian binary/
     apt-get update
apt-get install jenkins
     
   • Installs at http://localhost:8080/ by default
   • Followed steps to configure Jenkins to run behind Apache  https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache
   • configuration is in /etc/default/jenkins
   • I changed default port from 8080 for added security
   • add HTTP_HOST=127.0.0.1 #only takes requests from apache2
   • append to JENKINS_ARGS="... --prefix=/jenkins httpListenAddress=$HTTP_HOST"
   • To restart after config changes:
     /etc/init.d/jenkins force-reload
   • Log file will be placed in /var/log/jenkins/jenkins.log. Check this file if you are troubleshooting Jenkins.
   
   
2. Setting up apache for Jenkins
   Apache will be a reverse proxy to the Jenkins web application. We've already set up Jenkins to only accept requests from Apache (localhost), we will also add basic authentication. See also (https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu)
   
   • Create or copy this file called "jenkins" to /etc/apache2/sites-available
     <VirtualHost *:80>
ServerAdmin your.email@yourorganization.com
ServerName your.ip.here.goes.here
ServerAlias debian-build
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPreserveHost on
ProxyPass /jenkins http://localhost:8089/jenkins
ProxyPassReverse /jenkins http://localhost:8089/jenkins
<Location /jenkins/>
AuthType basic
AuthName "Jenkins"
AuthUserFile "/etc/apache2/sites-available/jenkins.htpasswd"
Require valid-user
</Location>

   • Run the following commands to set up Apache for virtual hosts and create the site "/jenkins"
     a2enmod proxy
a2enmod proxy_http
a2enmod vhost_alias
a2ensite jenkins
apache2ctl restart

     
   • Note: apache logs under /var/log/apache/error.log
   • I am no apache security configuration expert, but following this article was useful for configuring basic authentication for now. https://wiki.jenkins-ci.org/display/JENKINS/Apache+frontend+for+security
   • E.g in my case it was something like this: htpasswd -cm /etc/apache2/sites-available/jenkins.htpasswd your-username
   • In my opinion security is often overlooked in small scale projects, other general, but useful Apache security tips: http://httpd.apache.org/docs/2.0/misc/security_tips.html
   • After all of this you should be able to access Jenkins with the specified username/password at a URL like
   • http://your.ip.here.goes.here/jenkins
   
   
3. Install checkstyle
   For CheckStyle analysis of Java code (assuming you've already configured your project and build.xml to run CheckStyle during the build)
• Download checkstyle-5.3-bin.tar.gz (at the time of this post 5.4 is available)
• Copy to somewhere on the server e.g. /usr/share
• Extract the tar
  $/usr/share# tar zxf check checkstyle-5.3-bin.tar.gz
• Log in to Jenkins and navigate to Manage Jenkins > Configure System
• Check "Environment Variables" under Global Properties
• Add name = "CHECKSTYLE_HOME" value = "/usr/share/checkstyle-5.3"
  
4. Install findbugs
   Similarly for FindBugs analysis of Java code. (* I couldn't actually get this to successfully run on the EC2 Micro instance, it appeared to make the server run out of memory and the Jenkins build process would crash.)
• Download findbugs-1.3.9.tar.gz
• Copy to somewhere on the server e.g. /usr/share
• Extract the tar
  $/usr/share# tar zxf check findbugs-1.3.9.tar.gz
• Log in to Jenkins and navigate to Manage Jenkins > Configure System
• For "Environment Variables" under Global Properties
• Add name = "FINDBUGS_HOME" value = "/usr/share/findbugs-1.3.9"
• Copy the findbugs_ant.jar to ANT_HOME/lib
• $/usr/share/findbugs-1.3.9/lib# cp findbugs-ant.jar /usr/share/ant/lib
5. Manage Jenkins plugins for Static Analysis
   
   • Log in to and navigate to Jenkins > Manage > Available >
   • Check the following plugins
   • Static Analysis Utilities
   • Checkstyle Plug-in
   • FindBugs Plug-in
   • etc...
   • Jenkins may need to be restarted manually using /etc/init.d/jenkins force-reload
   
   
6. Configure Jenkins project: PunchClock
   
   • Log in to Jenkins and click New Jobs from the left menu. Fill out the following form. The details will vary per your project and needs.
   • Job Name: PunchClock
   • Set repostitory to Subversion
   • URL: http://ip.of.my.svn.repo/svn/trunk/TimeTrack
   • Will fail, need to enter credentials
   • Add Build Step > Invoke Ant
   • Check publishing options
   • Publish Checkstyle analysis results
   • Publish FindBugs analysis results
   • Publish combined analysis results
   • Publish JUnit test result report
   • My Test Result XMLs use the pattern "TEST*.xml"
   • Archive the artifacts
   • This depends on what artifacts your build.xml creates, my PunchClock complete packages are zipped in the distributions folder of my work space so: Files to archive: "distributions/*.*"
   • Optional you can set an automated checkout strategy. Under Build Triggers I chose "Poll SCM"
   • For schedule I used "@hourly"

Amazon EC2 Configuration

This is a whole other technology, which deserves its own post. I'll touch on what I did find in the end. I was motivated to look into this, Amazon AWS recently began offering the ability to have a single "micro" sized instance hosted for free. This gets you among other things, a virtual machine with 620 MB of memory, a single "virtual cpu", about up to 10GB EBS storage space, the OS of your choice* and a certain monthly bandwidth. I haven't been on the service a month yet, so we'll see how I get billed.

*The idea with these instances is that they are virtual machines that you can start from a pre-packaged snapshot referred to as an AMI. Amazon provides some (of their own linux flavor) but most appear to be community created public images for different OS's. I found a lot of information around this whole topic, as well as really well done, standardized AMI's from http://alestic.com/, which focuses mostly on Ubuntu images. The EC2StartersGuide for Ubuntu is also a great starting resource, especially for managing and configuring your EC2 instances from a Ubuntu desktop. In my case with Debian, the ec2debian google group was also a good place to start.

Long story short, when you sign up for AWS and log in to the Management Console, go to "Instances" and click on Launch Instance, you can search for an select a starting AMI. I went wit this one from the ec2debian group

us-east-1
i386 EBS: ami-0ce41865
x86_64 EBS: ami-80e915e9
i386 8gb instance: ami-c041bda9
x86_64 8gb instance: ami-1cbc4375


The distinction between instance and EBS is that instance root volume apparently doesn't survive a stoppage of the VM, not sure the benefit of that other than to keep things clean? I went with the x86_64 EBS: ami-80e915e9 after a false start with the 32 bit version.

Note these all start out with a 1GB Root volume. Needs to be reized.

After booting up, and figuring out how to connect to the instance (only can use ssh, scp...be prepared to use vi!).

Some issues I hit:
Java 6 won't install on 32-bit "micro" instance
Originally I tried a 32 bit debian instance AMI. Hit this bug everytime I installed sun-java6-jdk: https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/634487 which basically caused the whole instance to hang due to 100% CPU usage from a runaway install process. The only fix was to use a 64-bit AMI.

Enlarging EBS Root Volume
After finally installing everything and pressing the final key-stroke to start Jenkins, I was severly disappointed to only get a 503: Service Unavailable error from Apache. The Jenkins log wasn't created even and the Apache logs just stated that it couldn't connect to the Jenkins site. After banging my head against the wall, I found a not enough disk space error in the /var/logs/daemon log. It turned out I maxed the 1GB initial volume size. Unfortunately EBS volumes can't just be "dialed up" like you can do with a VMWare Player VM. Some guidance can be found here on how to resize a root EBS volume:  http://alestic.com/2010/02/ec2-resize-running-ebs-root

Basically what I ended up doing using the AWS Management Console

1. Create EBS snapshot
2. Stop Instance
3. Detach old volume
4. create new volume from snapshot
5. Attach new volume (as /dev/sda1...same as old volume!)
6. Start Instance
7. resize2fs /dev/xvda1 (Debian is tricky that it doesn't list the volume as /dev/sda1). It did an online resize in about 5 minutes and I was good to go.