On Configuring Automated Builds on an Amazon EC2 micro-instance using Jenkins, SVN and Apache

I am a big fan of the continuous integration approach with using tools and build scripts to automate the build process for programs. Combine this with comprehensive unit testing and static analysis provides some easy ways to improve the quality of code.

So I wanted to implement a continuous integration set up, partly as a way to improve my infrastructure around the PunchClock project, as well as a chance to explore some new technologies, especially Amazon Web Services.

What I settled on was to create a "build machine" using Jenkins CI, formerly Hudson, an open source program specifically for automating building software in a continuous integration environment. On the same server I would also host my Subversion repository (SVN), and being that it is an Amazon EC2 instance I have less worry of a possibility of a catastrophic loss of data from my current Ubuntu box at home. Both SVN and the Jenkins web application would sit behind an Apache2 Web Server. All of this would be on a Debian Linux distro running as an EC2 instance. (I chose to go with Debian purely to gain exposure to other distros besides Ubuntu). I learned a lot from this set up, the AWS/EC2 was actually more challenging than you would normally find from a hosted server, but their AWS Management Console allows you to do most operations pretty easily.


Prereqs

1. Install Apache2 if not present
   On the default Debian install Apache is present, but on the Debian EC2 AMI I used it wasn't installed.
   apt-get install apache2
   
2. Install Java 1.6 (update 25)
   You'll need to edit your /etc/apt/sources.list see http://jayraj87.blogspot.com/2010/02/installing-sun-java6-jdk-in-debian.html
   
3. Install Ant 1.8.0
   
   • installs version 1.8.0 to /usr/bin/ant (/usr/share/ant)
     apt-get install ant
     
   • copy junit.jar to ANT_HOME\lib (junit 3.8.2 or later download here)
4. Install SVN and Repository
   
   • I installed SVN and then moved my existing repository to a location (this is a good article on moving svn: http://linux.justinhartman.com/Moving_a_Subversion_repository_to_another_server) . 
   • Also configured Apache2 with dav_svn for webaccess and basic authentication (http://davidwinter.me/articles/2006/02/16/subversion-over-apache-2-on-ubuntu/)
5. Before you begin I assume you have your own working ANT script that would compile, run JUnit tests, run any static analysis utilities (e.g. FindBugs or CheckStyle) and package your project into artifacts for distribution. The PunchClock application had an existing ANT build.xml that I started from.
   

Installation

All of these steps I honed on a default Debian install. I tested this on VMWare Player running the 6.0.1a "Squeeze" Debian install, from an iso file found on BitTorrent. Once I had got the steps down, basically the same applied to installing on the Debian instance on Amazon EC2 except for a few issues. All of these steps were performed as root unless otherwise noted.

1. Install Jenkins LTS
   
   • 
     wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
     
     • Add site to debian packages /etc/apt/sources.list: deb http://pkg.jenkins-ci.org/debian binary/
     apt-get update
apt-get install jenkins
     
   • Installs at http://localhost:8080/ by default
   • Followed steps to configure Jenkins to run behind Apache  https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache
   • configuration is in /etc/default/jenkins
   • I changed default port from 8080 for added security
   • add HTTP_HOST=127.0.0.1 #only takes requests from apache2
   • append to JENKINS_ARGS="... --prefix=/jenkins httpListenAddress=$HTTP_HOST"
   • To restart after config changes:
     /etc/init.d/jenkins force-reload
   • Log file will be placed in /var/log/jenkins/jenkins.log. Check this file if you are troubleshooting Jenkins.
   
   
2. Setting up apache for Jenkins
   Apache will be a reverse proxy to the Jenkins web application. We've already set up Jenkins to only accept requests from Apache (localhost), we will also add basic authentication. See also (https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu)
   
   • Create or copy this file called "jenkins" to /etc/apache2/sites-available
     <VirtualHost *:80>
ServerAdmin your.email@yourorganization.com
ServerName your.ip.here.goes.here
ServerAlias debian-build
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPreserveHost on
ProxyPass /jenkins http://localhost:8089/jenkins
ProxyPassReverse /jenkins http://localhost:8089/jenkins
<Location /jenkins/>
AuthType basic
AuthName "Jenkins"
AuthUserFile "/etc/apache2/sites-available/jenkins.htpasswd"
Require valid-user
</Location>

   • Run the following commands to set up Apache for virtual hosts and create the site "/jenkins"
     a2enmod proxy
a2enmod proxy_http
a2enmod vhost_alias
a2ensite jenkins
apache2ctl restart

     
   • Note: apache logs under /var/log/apache/error.log
   • I am no apache security configuration expert, but following this article was useful for configuring basic authentication for now. https://wiki.jenkins-ci.org/display/JENKINS/Apache+frontend+for+security
   • E.g in my case it was something like this: htpasswd -cm /etc/apache2/sites-available/jenkins.htpasswd your-username
   • In my opinion security is often overlooked in small scale projects, other general, but useful Apache security tips: http://httpd.apache.org/docs/2.0/misc/security_tips.html
   • After all of this you should be able to access Jenkins with the specified username/password at a URL like
   • http://your.ip.here.goes.here/jenkins
   
   
3. Install checkstyle
   For CheckStyle analysis of Java code (assuming you've already configured your project and build.xml to run CheckStyle during the build)
• Download checkstyle-5.3-bin.tar.gz (at the time of this post 5.4 is available)
• Copy to somewhere on the server e.g. /usr/share
• Extract the tar
  $/usr/share# tar zxf check checkstyle-5.3-bin.tar.gz
• Log in to Jenkins and navigate to Manage Jenkins > Configure System
• Check "Environment Variables" under Global Properties
• Add name = "CHECKSTYLE_HOME" value = "/usr/share/checkstyle-5.3"
  
4. Install findbugs
   Similarly for FindBugs analysis of Java code. (* I couldn't actually get this to successfully run on the EC2 Micro instance, it appeared to make the server run out of memory and the Jenkins build process would crash.)
• Download findbugs-1.3.9.tar.gz
• Copy to somewhere on the server e.g. /usr/share
• Extract the tar
  $/usr/share# tar zxf check findbugs-1.3.9.tar.gz
• Log in to Jenkins and navigate to Manage Jenkins > Configure System
• For "Environment Variables" under Global Properties
• Add name = "FINDBUGS_HOME" value = "/usr/share/findbugs-1.3.9"
• Copy the findbugs_ant.jar to ANT_HOME/lib
• $/usr/share/findbugs-1.3.9/lib# cp findbugs-ant.jar /usr/share/ant/lib
5. Manage Jenkins plugins for Static Analysis
   
   • Log in to and navigate to Jenkins > Manage > Available >
   • Check the following plugins
   • Static Analysis Utilities
   • Checkstyle Plug-in
   • FindBugs Plug-in
   • etc...
   • Jenkins may need to be restarted manually using /etc/init.d/jenkins force-reload
   
   
6. Configure Jenkins project: PunchClock
   
   • Log in to Jenkins and click New Jobs from the left menu. Fill out the following form. The details will vary per your project and needs.
   • Job Name: PunchClock
   • Set repostitory to Subversion
   • URL: http://ip.of.my.svn.repo/svn/trunk/TimeTrack
   • Will fail, need to enter credentials
   • Add Build Step > Invoke Ant
   • Check publishing options
   • Publish Checkstyle analysis results
   • Publish FindBugs analysis results
   • Publish combined analysis results
   • Publish JUnit test result report
   • My Test Result XMLs use the pattern "TEST*.xml"
   • Archive the artifacts
   • This depends on what artifacts your build.xml creates, my PunchClock complete packages are zipped in the distributions folder of my work space so: Files to archive: "distributions/*.*"
   • Optional you can set an automated checkout strategy. Under Build Triggers I chose "Poll SCM"
   • For schedule I used "@hourly"

Amazon EC2 Configuration

This is a whole other technology, which deserves its own post. I'll touch on what I did find in the end. I was motivated to look into this, Amazon AWS recently began offering the ability to have a single "micro" sized instance hosted for free. This gets you among other things, a virtual machine with 620 MB of memory, a single "virtual cpu", about up to 10GB EBS storage space, the OS of your choice* and a certain monthly bandwidth. I haven't been on the service a month yet, so we'll see how I get billed.

*The idea with these instances is that they are virtual machines that you can start from a pre-packaged snapshot referred to as an AMI. Amazon provides some (of their own linux flavor) but most appear to be community created public images for different OS's. I found a lot of information around this whole topic, as well as really well done, standardized AMI's from http://alestic.com/, which focuses mostly on Ubuntu images. The EC2StartersGuide for Ubuntu is also a great starting resource, especially for managing and configuring your EC2 instances from a Ubuntu desktop. In my case with Debian, the ec2debian google group was also a good place to start.

Long story short, when you sign up for AWS and log in to the Management Console, go to "Instances" and click on Launch Instance, you can search for an select a starting AMI. I went wit this one from the ec2debian group

us-east-1
i386 EBS: ami-0ce41865
x86_64 EBS: ami-80e915e9
i386 8gb instance: ami-c041bda9
x86_64 8gb instance: ami-1cbc4375


The distinction between instance and EBS is that instance root volume apparently doesn't survive a stoppage of the VM, not sure the benefit of that other than to keep things clean? I went with the x86_64 EBS: ami-80e915e9 after a false start with the 32 bit version.

Note these all start out with a 1GB Root volume. Needs to be reized.

After booting up, and figuring out how to connect to the instance (only can use ssh, scp...be prepared to use vi!).

Some issues I hit:
Java 6 won't install on 32-bit "micro" instance
Originally I tried a 32 bit debian instance AMI. Hit this bug everytime I installed sun-java6-jdk: https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/634487 which basically caused the whole instance to hang due to 100% CPU usage from a runaway install process. The only fix was to use a 64-bit AMI.

Enlarging EBS Root Volume
After finally installing everything and pressing the final key-stroke to start Jenkins, I was severly disappointed to only get a 503: Service Unavailable error from Apache. The Jenkins log wasn't created even and the Apache logs just stated that it couldn't connect to the Jenkins site. After banging my head against the wall, I found a not enough disk space error in the /var/logs/daemon log. It turned out I maxed the 1GB initial volume size. Unfortunately EBS volumes can't just be "dialed up" like you can do with a VMWare Player VM. Some guidance can be found here on how to resize a root EBS volume:  http://alestic.com/2010/02/ec2-resize-running-ebs-root

Basically what I ended up doing using the AWS Management Console

1. Create EBS snapshot
2. Stop Instance
3. Detach old volume
4. create new volume from snapshot
5. Attach new volume (as /dev/sda1...same as old volume!)
6. Start Instance
7. resize2fs /dev/xvda1 (Debian is tricky that it doesn't list the volume as /dev/sda1). It did an online resize in about 5 minutes and I was good to go.

Introducing the Time Track Project

The Time Track application is a small side-project I have been developing since late November 2010. It is a simple application that lets a user keep track of time spent on work, projects, tasks etc.

Time Track main window, Windows Vista x64

The main requirements for the Time Track application were

• Be a software version of a time punch clock, for any purpose


• Keep track of time in a persistant format that can be saved


• Cross-platform support (at least Windows, Mac OS X)


• Needs a GUI


• Lightweight, simple to use and set up

I developed this in Java as a standalone desktop program with a GUI using all OpenSource components. I am nearly complete with an initial version that meets all this functionality and I intend to release it as a free-ware app for now, maybe it can be made released as open source by itself in the future.

Implementation

I had really two paths I could have taken with this, one would be to build a dynamic webapp with a RDBMS backend. The second approach was to develop a standalone self-contained application that could be somewhat portable itself. I was hesitant to take on a project that required any sort of server set up and hosting, I am just not set up to do that now, so although the webapp was pretty easily cross-platform being browser based, I didn't like the idea of the dependency on the remote server.

I went with the standalone app approach, in order to be cross-platform and easy for me to develop I went with Java as the source language. I didn't want the program looking like a Java Swing application so I opted for the SWT api which is the basis for the Eclipse IDE and platform, it tends to look more "solid" in my opinion as one of its main features is native look and feel for UI components on every OS. I think so far it is coming along nicely. You can see a side-by-side comparison of the interface in Windows Vista, and on Linux (Ubuntu 10.04, 64-bit)

I have probably spent around 5 weekends and the occasional hour on a weeknight in total since I started building this application. Probably overall time spent has been under 80 hours. It was enjoyable to more or less go through the full development life-cycle for a new project. I got to do design, open-source research, coding and integration, source-control, and complete build scripts in Ant. Even though I may never release a line of code from this, I am building like I could.

In terms of the OpenSource projects that I used a variety of different components in my stack.

• http://www.eclipse.org/swt - Standard Widget Toolkit for all User Interface
• http://db.apache.org/derby - Apache Derby for lightweight embedded relational database connected with JDBC
• http://sourceforge.net/projects/opencsv - OpenCSV library for creation of CSV format exports of data
• http://tango.freedesktop.org/Tango_Desktop_Project - library of free icons

For my build environment, I am using Java JDK 1.6.0_17, Eclipse Ganymede IDE, Junit 3 (for the few automated unit tests that were really helpful with Derby), SVN for source control and builds are done with ANT. Testing was done on Windows Vista SP2 x64 and Ubuntu Linux 64-bit GTK.

RoadMap
On my to do list I still have a number of items I'd like to close out before really wrapping up this release.

• The main table needs to update when changes occur. Currently it is loaded at start up.
• Need to think about an import data function.
• Testing on a Mac
• Implement a more solid connection pool mechanism
• Release strategy. Post on this site? or create an OpenSource project page?

Subclipse SVN plug-in for Eclipse

10/10/10 - had to post something.

I've experiences utter frustration with the SVN command-line client to maintain the various projects I've been working on and the lack of a Linux equivalent to TortoiseSVN. Typical example, I added a directory to the repository by mistake before running a clean operation. It took 20 minutes trying to delete files, cleanup commands to get back to a starting place with the files I really wanted to add.

The only apparent GUI for SVN on Linux appears to be through an Eclipse plug-in. Since I am moving my projects to Eclipse anyway this is a no-brainer.

Subclipse

Subclipse is the Eclipse client that integrates directly with the Eclipse IDE, even my Eclipse CDT IDE works. The download instructions are out of date with the Helios package of Eclipse (or Linux version). But you can more or less follow the instructions they give.

The installation is performed from within Eclipse

1. Start Eclipse
2. Help > Install New Software
3. In the Install wizard, Work with: field, enter the URL for the Subclipse version from the download and install page of the Subclipse site. In my case: http://subclipse.tigris.org/update_1.6.x.
4. I selected all the packages there, Core SVNKit Library, Optional JNA Library, Subclipse.
5. Finish your way through the wizard to install everything.
6. After the install you have to restart Eclipse.
7. I got a JavaHL error on start up of Eclipse, having to do with the JNA library. This is another topic discussed on the Subclipse wiki.
8. I ignored the error and Subclipse worked so I don't know how serious it is. I since went and installed then performed the JavaHL test cases successfully.
9. Note: I already had the SVN subversion command-line client installed, that may be a prerequisite, I am not sure to be honest.


The usage of Subclipse is not that clear-cut and documentation is lacking.

1. You need to set up a repository. Under Window > Open Perspective > Other choose SVN Repository Exploring. You can add a new repository if you have the URL (I had already created a local repository with the command line client, "file:///home/ammianus/svn-repo").
2. From the SVN Repository Explorer perspective, you can right-click a folder in your repository then click Import..., select a project in your workspace and you are good to go.
3. Once a project is associated with a location in your repository, if you right-click on a project or file > Context Menu > Team > * there will be all the normal SVN commands you would need. You will also notice that next to the project name will be the SVN repository location in brackets [].
4. You can see the behavior of Subclipse in a Console view in which you can switch to the SVN console output.