Multi-Stage Continuous Integration with AccuRev
Learn how an AccuRev customer uses MSCI
» read blog post
Large Scale Multi-Stage Continuous Integration in
Real-Time
» watch webinar
Until now, Agile and continuous integration have not scaled well beyond small, co-located teams. In a typical project, the integration phase is towards the end and is the source of a great deal of problems and delays. One practice that has emerged to address this problem is continuous integration. Continuous integration enables development teams to accelerate their development lifecycle through shorter iterations, enabling software builds upon source code promotions from the software configuration management (SCM) product.
Multi-stage Continuous Integration isolates individual developer’s code and only builds on working and fully tested code bases. This mitigates the risk within larger development teams where the dependencies required for build are generally not visible, correlated or tested. Teams continue at maximum productivity, even while their projects are being built and tested.
The basis of Multi-Stage Continuous Integration is this:
If individual isolation is a good idea, then isolation for features, teams, team integration, staging, QA and release is an even better idea.
Multi-stage Continuous Integration (MSCI) is an extension of the common practice of shielding others from additional changes by only checking-in when individual changes have been tested and only updating an individual workspace when it’s ready to absorb other people’s changes. With MSCI, each team does a team-based continuous integration first and then cross-integrates the team’s changes with the mainline on success. This limits project-wide churn and allows continuous integration to scale to large projects.
“I’ve seen Multi-Stage Continuous Integration successfully implemented in many shops and every time the developers say something like: “I never realized how many problems were a result of doing mainline development until they disappeared.“
- Damon Poole, AccuRev CTO
When creating software, you want to design it so that it is flexible and adapts to new circumstances. The same should be true for your development organization. MSCI is designed to solve the problem of scaling Continuous Integration development efforts when teams grow, or are geographically distributed.
MSCI takes advantage of a basic unifying pattern of software development: software moves in stages from a state of immaturity to a state of maturity, and the work is broken down into logical units performed by interdependent teams that integrate the different parts together over time. What changes from shop to shop is the number of stages, the number and size of teams, and the structure of the team interdependencies.
When a team is in the process of integrating the work of its team members, it does not need to be disrupted by the changes from other teams and conversely, it would be better for the team not to disrupt other teams until they have integrated their own work. But just as is the case with the individual, there should be continuous integration at the team level, but then also between the team and the mainline.
This diagram shows a hierarchy of branches with changes flowing from top to bottom and in some cases back towards the top. Each box graphs the stability of a given branch in the hierarchy over time. At the top are individual users. They are making changes all day long. Sometimes their work areas build, sometimes they don’t. Sometimes the tests pass, sometimes they don’t. Their version of the software is going from stable to unstable on a very frequent basis, changing on the order of every few minutes. Hopefully, users only propagate their changes to the next level in the development hierarchy when the software builds for them and an appropriate amount of testing has been done. That happens on the order of once per hour or so, but ideally it happens no less than once per day.
Then, just as individuals check-in their changes when they are fully integrated, the team leader will integrate with the next level and when the integration build and test are done they will merge the team’s changes to the next level. Thus, team members see each other’s changes as needed, but only team member’s changes. They see other team’s changes only when the team is ready for them. This happens on the order of several times per week and perhaps even daily.
Changes propagate as rapidly as possible, stopping only when there is a problem. Ideally, changes make it to the main integration area just as frequently as when doing mainline development. The difference is that fewer problems make it all the way to the main integration area. Multi-Stage Continuous Integration allows for a high degree of integration to occur in parallel while vastly reducing the scope of integration problems.
It takes individual best practices that we take for granted and applies them at the team level.