Litle & Co. Accelerates Agile XP Software Development Process with AccuRev

Success Highlights
Litle

Users and Environment

  • More than 30 developers and QA people working at the same location
  • Agile/XP software development since 2001
  • A codebase that contains 50,000 files, including test cases
  • Monthly iteration cycle
  • Twelve overlapping iterations per year

AccuRev Success Highlights

  • AccuRev enabled Litle to easily scale its Agile/XP environment
  • Multi-Stage Frequent Integration reduced build failures by 50 percent
  • AccuRev eliminated the need to use labels, saving an hour off each build
  • AccuRev eliminated two days of downtime each month by eliminating the single huge merge at the end
  • AccuRev simplified the process of passing PCI and SAS70 Type 2 audits

Customer Profile

Litle & Co. is one of few payment processing companies focused exclusively on serving direct-to-consumer merchants. The company’s new technology and fresh approach to the development of payment systems have resulted in a next-generation payment processing platform that provides more services to merchants while simultaneously reducing their maintenance costs and giving them more flexibility and options.

Founded in 2001, Litle & Co. has won a number of prestigious awards, including the No. 1 ranking on Inc. magazine’s 2006 list of the 500 fastest-growing private companies in the United States.

Background

At the request of developers and with the blessing of its founder and chairman since its founding Litle has employed many of the practices associated with Extreme Programming (XP), including pair programming, Prior to deploying AccuRev software in early 2007, Litle & Co. was using CVS to manage its Agile software projects.

The Team

Development teams at Litle are entirely collocated, with no offshore or off-site developers. For each project, the company assigns a team consisting of multiple pair programming teams. Over time, the development team has grown to a team of more than 30 members, including QA and a codebase that contains 50,000 files including test cases. For this purpose, the team has had to add additional practices in order to scale XP.

Product Development Challenges

Since Litle & Co. interfaces with the world’s major credit card and alternative payment networks, such as Visa, MasterCard, and PayPal, its mission-critical applications must be available 24/7, with no downtime. As the development team grew, it became increasingly challenging to manage its Agile software development process using CVS. As Litle put it, they “hit the knee in the curve” and knew that it was time for a more efficient solution to manage its Agile development environment and monthly iteration cycle.

Litle’s software is hosted at multiple off-site data centers, and the company upgrades its software every month like clockwork by using approximately six-week iterations, some of which overlap in order to release monthly. The first four weeks of the iteration are focused on development, including design, estimating, unit testing, integration testing, and QA. The final two weeks are a dry run of the install process and a training period on the changes in the system for the production staff. While the final two-week part of the process is taking place, the development team has already started the next iteration, resulting in a two-week overlap. There are a total of 12 iterations a year.

These overlapping iterations were one of the major reasons that CVS wasn’t scaling. People generally avoid branches in CVS because of the pain associated with merging back into mainline development. Litle now uses a new AccuRev stream every month in order to isolate changes between the overlapping iterations as well as to keep each iteration up to date with the most recent release. The developers are also able to easily integrate those changes and coordinate the propagation of features between any two iterations.

“You just can’t cut a branch in CVS like you can instantly create a stream in AccuRev. With AccuRev, you don’t have to predict the future or worry where you’re going to put it. You’re not worried about what work is going to be completed first for your merge strategy since you’re constantly performing incremental merges, and you’re never stuck with your decision. AccuRev is much more agile that way."

– David Tarbox, Director, Software Development

Agile stresses a constant “build and integration” type of process, yet Litle’s development organization found it easiest to hold off merging code changes in CVS until the end of a project. By continuing to use this one-branch method, merging was costing the company as much in time and money on its test data as on its code. Labeling a build-alone was taking an hour each time. Litle also wanted more segregation of projects, teams, and people as a way to isolate merges and builds, but realized that the merge costs would be too high using CVS.

“Obviously, coming from CVS, we looked at Subversion, but it didn’t solve the underlying architectural branch-based shortcomings. The AccuRev stream-based inheritance model allows us to do lightweight, incremental merges and track their history throughout the software development life cycle—both up and down the structure—automatically. We would have to do that manually in another tool. The biggest surprise was that AccuRev has allowed us to do things we couldn’t conceive of with CVS—that has resulted in a greater benefit to the company than all the efficiency gains over CVS.”

The biggest problem Litle development teams ran into as they grew was integration of newly developed code into the existing codebase. To address this, in 2007, the company added the practice of Multi-Stage Frequent Integration to its implementation of XP. The teams do frequent (twice daily) integration instead of continuous integration because a full build/test cycle takes five hours.

Solution: Multi-Stage Frequent Integration

With AccuRev, Litle project teams are now segregated using AccuRev streams, and each team performs frequent builds. Using Multi-Stage Frequent Integration, each project works against its own team stream, merging changes from the mainline into its team stream every day or two, and doing its own build/test cycle with just that team’s changes.

When the build/test cycle is successful, the team then merges its changes into the mainline. As a result of moving to Multi-Stage Frequent Integration, any team-specific problems are identified and removed prior to mainline integration, and the only problems that arise in the mainline are those that are due to interactions between changes from multiple teams. The isolation also simplifies the job of figuring out what changes may have caused a problem because the project lead only has to look at changes made by a single team, not the changes made by all developers.

AccuRev can also display all changes made between two specific periods of time, narrowing further the possible location of any build failure.

Prior to implementing Multi-Stage Frequent Integration, teams would have to manually pore over all build and test failures to determine which change caused which failures. This was done by very senior developers who had to be familiar with all aspects of the system so that they could understand complex interactions between unrelated components. With CVS, this window of failure could be as great as 24 hours, resulting in a tremendous loss of productivity.

“AccuRev’s object-oriented inheritance model takes all the error-prone manual scripting and time-consuming branch creation out of the equation. AccuRev was the only solution that could have made the implementation of this continuous integration practice elegant, straightforward, and error free.”

Satisfying Audits with Integrated Issue Tracking

Tests and code are written in parallel for each work item as that work item is started. All assigned work is tracked in AccuRev, and all work that developers do is associated with a work item in AccuRev’s integrated issue tracking system. This is done to automate compliance because, as a credit card processing company, Litle needs to pass biannual SAS70 Type 2 audits and annual Payment Card Industry (PCI) audits. When Litle used CVS, much of this had to be done manually.

“That’s another thing we like about AccuRev: if you want to do software development the right way, the tool is not going to get in your way. In fact, it’s kind of set up so that if you’re doing all the right stuff, the tool fits into your toolbox quite nicely.”

Litle also sees its long track record of successful use of Agile as a competitive advantage: because they are on monthly releases, salespeople are able to work with the product people to adjust the company’s priorities to accommodate what makes the most sense for the business. Rather than having to pick everything the company needs to fit in for the year now and then get together again in a year, prioritization on all occurs during a regular monthly meeting where they look at the priorities together. This is important to help the company pursue any natural expansion of its business in the shortest business cycles.

For additional information regarding this success story or to leave comments, visit Damon Poole’s Agile Development Thoughts blog at http://damonpoole.blogspot.com/2008/05/agile-case-study-litle-co.html.

icon: pdf Download the Case Study