Continuous Integration and Benchmark Monitoring

Introduction

Developing high-quality software is a challenging task, especially when it is done by a (distributed) team of programmers. In the last decade, several concepts and tools have gained acceptance in the industry that can raise the quality of software significantly, for example, automated test suites, code versioning systems, issue trackers and nightly builds.

The idea behind Continuous Integration is to integrate all these concepts into a single (extensible) tool that continuously applies quality control measures while the software is being developed. For example, whenever a developer commits changes to the code base, the CI tool runs the test suite and reports the results - including statistics such as code coverage - to a website. For open source projects, such websites are often publicly available and can be used to estimate the stability of a given application. If the project builds successful then a snapshot can be published, if tests fail then the developer who committed the changes can be notified.

CI tools are widely accepted in the industry and used by companies such as Google, Intel, Pixar, Credit Suisse, Mercedes-Benz [Zutubi, Bamboo] or the Ergon Informatik AG.

Current situation

We currently use our own build tool (written in Python) to perform a sort of poor-man's continuous integration, including sending notifications to involved developers. The tool works reasonably well, but we would like to migrate to an established CI tool since it appears to be the better choice in the long run with respect to maintainability and already implemented features. However, we have a few requirements such as tracking and evaluating arbitrary statistics that are probably not supported out of the box, and thus have to be added to the CI tool as plug-ins or extensions.

Challenges

• Our test suite and the set of available statistics are both not constant, that is, tests and statistics can be added or removed over time. This raises the question if a classical SQL database with a fixed schema is the right tool for recording test results and statistics, or if a NoSQL database, more specifically, a document-oriented database, is the better choice.
• A related challenge is finding a sweet spot for the trade-off between flexibility and type-safety. A fixed database schema can be easily mapped to a class with fields of concrete types (String, Int, ...), whereas data with an arbitrary structure yields a map/dictionary with String keys and Object values in the worst case.
• Jenkins CI looks like an interesting candidate for a CI tool. It is written in Java, so it should be possible to write plug-ins in Scala, which (currently) is our preferred programming language, and which compiles to JVM byte code. However, according to the mailing list, integrating non-Java-plug-ins is not always straight-forward.
  

Opportunities

• Gain experience with industry-strength quality assurance measures
• Work with Scala, a modern language combining object-oriented and functional programming with a powerful type-system
• Work with non-relational databases
  
• Gain experience with the inter-operability of JVM languages
  

Recommended background

• Interest in software engineering and web development
• Experience with an object-oriented language
  
• Experience with databases
  

Contact

Malte Schwerhoff