Goal

To create a crossplatform, distributed continuous integration system for open simulator grids which can automate both builds and testing.

People

The list of people who are working to develop a solution:

• Daedius Moskvitch (daedius @@@@@ daedius.com)

Features

• Crossplatform
• Notification via email,irc,web
• Master-Slave architecture
• Nant support
• NUnit support
• Support for grid testing

Released Versions

0.1

Released: January 1st, 2008
A simple application that can detect changes on the opensim svn, attempt compile, and return whether it was successful or not

• Sources

Features

• svn source control support
• svn change detection
• linux and windows support

Installation

Right now there is nant build files and visual studio 2005 solution.

Special Notes For Running

After compiling the application, make sure you have the correct application configuration file for your operating system (the current default is Windows, it will fail on Linux if you do not change). Just rename the right App.config to "BuildBot.exe.config".

Other Notes

• Deletion can be problematic on windows
• I think TortoiseSVN on win32 causes alot of trouble when deleting source control directories
• Need more user friendly exceptions
• A good way to force building, is to change the LatestTime.txt to a few days back ( to gaurantee that new changes are detected)
• HAPPY NEW YEARS!

Contributors

• Daedius Moskvitch (daedius @@@@@ daedius.com)

0.2

Released: January 5th, 2008
A simple application that can detect changes on the opensim svn, multithread a build process (or multiple build processes), distribute nant files locally, attempt compile, and return status

• Sources

Features

Create a more configurable build runner via xml. This release will hopefully be a good precursor to having a real operational slave.

• Interfacing of master-slave architecture
• Linux, Windows, and OSX support
• Initial slave configuration in external xml
• Working slave that can multi-thread builds from masters
• Simple status and result chain in builds run on slaves
• Nant-based builds
• Custom nant tags for notification of status within a nant process (i.e. <status value="Running" Description="Compiling OpenSim" buildid="${build.id}"/> )

Other Notes

• Be sure to copy the write App.Config to BuildBot.Net.exe.config, and set the argument BuildContainer equal to the path of the directory "BuildContainer"

Contributors

• Daedius Moskvitch (daedius @@@@@ daedius.com)

Planned Versions

0.3

Create a real master slave architecture to distribute our nant builds and retrieve statuses in realtime

• Build master to distribute nant files to build slave
• Simple status plugin
• Working status chain

0.4

Finally get some results that are viewable to the public!

• Create a website that uses javascript to display build statuses in realtime

Architecture Plans

The Buildbot consists of a single buildmaster and one or more buildslaves, connected in a star topology. The buildmaster makes all decisions about what, when, and how to build. It sends commands to be run on the build slaves, which simply execute the commands and return the results. (certain steps involve more local decision making, where the overhead of sending a lot of commands back and forth would be inappropriate, but in general the buildmaster is responsible for everything).

The buildmaster is usually fed Changes by some sort of version control system (see Change Sources), which may cause builds to be run. As the builds are performed, various status messages are produced, which are then sent to any registered Status Targets (see Status Delivery).

The buildmaster is configured and maintained by the “buildmaster admin”, who is generally the project team member responsible for build process issues. Each buildslave is maintained by a “buildslave admin”, who do not need to be quite as involved. Generally slaves are run by anyone who has an interest in seeing the project work well on their favorite platform.

Build Slave Connections

The buildslaves are typically run on a variety of separate machines, at least one per platform of interest. These machines connect to the buildmaster over a TCP connection to a publically-visible port. As a result, the buildslaves can live behind a NAT box or similar firewalls, as long as they can get to buildmaster. The TCP connections are initiated by the buildslave and accepted by the buildmaster, but commands and results travel both ways within this connection. The buildmaster is always in charge, so all commands travel exclusively from the buildmaster to the buildslave.

To perform builds, the buildslaves must typically obtain source code from a CVS/SVN/etc repository. Therefore they must also be able to reach the repository. The buildmaster provides instructions for performing builds, but does not provide the source code itself.

Buildmaster Architecture

The Buildmaster consists of several pieces:

   * Change Sources, which create a Change object each time something is modified in the VC repository. Most ChangeSources listen for messages from a hook script of some sort. Some sources actively poll the repository on a regular basis. All Changes are fed to the Schedulers.
   * Schedulers, which decide when builds should be performed. They collect Changes into BuildRequests, which are then queued for delivery to Builders until a buildslave is available.
   * Builders, which control exactly how each build is performed (with a series of BuildSteps, configured in a BuildFactory). Each Build is run on a single buildslave.
   * Status plugins, which deliver information about the build results through protocols like HTTP, mail, and IRC. 

Each Builder is configured with a list of BuildSlaves that it will use for its builds. These buildslaves are expected to behave identically: the only reason to use multiple BuildSlaves for a single Builder is to provide a measure of load-balancing.

Within a single BuildSlave, each Builder creates its own SlaveBuilder instance. These SlaveBuilders operate independently from each other. Each gets its own base directory to work in. It is quite common to have many Builders sharing the same buildslave. For example, there might be two buildslaves: one for i386, and a second for PowerPC. There may then be a pair of Builders that do a full compile/test run, one for each architecture, and a lone Builder that creates snapshot source tarballs if the full builders complete successfully. The full builders would each run on a single buildslave, whereas the tarball creation step might run on either buildslave (since the platform doesn't matter when creating source tarballs). In this case, the mapping would look like:

    Builder(full-i386)  ->  BuildSlaves(slave-i386)
    Builder(full-ppc)   ->  BuildSlaves(slave-ppc)
    Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc)

and each BuildSlave would have two SlaveBuilders inside it, one for a full builder, and a second for the source-tarball builder.

Once a SlaveBuilder is available, the Builder pulls one or more BuildRequests off its incoming queue. (It may pull more than one if it determines that it can merge the requests together; for example, there may be multiple requests to build the current HEAD revision). These requests are merged into a single Build instance, which includes the SourceStamp that describes what exact version of the source code should be used for the build. The Build is then randomly assigned to a free SlaveBuilder and the build begins.

Status Delivery Architecture

The buildmaster maintains a central Status object, to which various status plugins are connected. Through this Status object, a full hierarchy of build status objects can be obtained.

The configuration file controls which status plugins are active. Each status plugin gets a reference to the top-level Status object. From there they can request information on each Builder, Build, Step, and LogFile. This query-on-demand interface is used by web plugin to create the main status page each time a web browser hits the main URL.

The status plugins can also subscribe to hear about new Builds as they occur: this is used by the MailNotifier to create new email messages for each recently-completed Build.

The Status object records the status of old builds on disk in the buildmaster's base directory. This allows it to return information about historical builds.

There are also status objects that correspond to Schedulers and BuildSlaves. These allow status plugins to report information about upcoming builds, and the online/offline status of each buildslave

Comparison To Other Automated Build/Testing Methods

CruiseControl.Net

• Supporting the master slave architecture will be vastly different from CC's centralized approach, but it will allow us more flexability and decentralization of tests on many platforms. It will also enable us to run certain tests which require multiple computers for grid testing.
• This project has a great system of xml specification for projects and support for Nant,Nunit,NCop, etc we could learn from thoug

CruiseControl

• Once again, our project will have a focus on distributed testing and building.
• Our support for C# technologies will be within our control if something goes wrong. Also, knowing how mono works on all platforms is much easier than knowing how both Java and Mono work on all platforms.

BuildBot

• Buildbot has a great architecture for distributed builds and testing which we will be using extensively as a general direction
• We will not use Python as a configuration language, rather we will be using XML structures more similar in CruiseControl
• We will have more direct support for C# technologies

Manual Building & Testing

• More efficient
• More automated
• More proven