Managing Vendor Code Customizations with AccuRev's Stream-based SCM

Why Streams Are Easier than Traditional Branches



Download PDF

photo: David Thomasby David Thomas
Senior Systems Engineer
AccuRev, Inc.

 

 


Summary

Customizing or extending third party “vendor” source code is becoming increasingly common especially with the availability of open-source software. Building upon existing code increases your time to market and lets a group of experts elsewhere develop the foundation. Vendors typically provide frequent patches and new features in the form of vendor releases. Managing the incorporation of vendor releases alongside customizations requires an additional layer of configuration management. Traditional branch-based software configuration management (SCM) tools require an unnecessarily complex branch and merge process. This article describes how AccuRev’s stream-based SCM provides a more intuitive and efficient parallel development model for managing customizations to vendor code.

The Challenge

Managing customizations to vendor code requires an additional layer of configuration management to integrate subsequent vendor releases. Vendor code must be imported, merged with the previously imported vendor release1, merged with a selected set of compatible customizations, and finally merged with one or more active codelines. The challenge is to independently track vendor code and orchestrate selective merging of custom features with vendor upgrades without jeopardizing or disrupting active codelines.

Why Traditional Branch Models Are Difficult

Traditional branch-based SCM models utilize numerous branches to track in parallel both vendor source and custom modifications. In a typical branch model, mainline represents the central development codeline2, a single vendor branch off of mainline isolates and tracks only vendor code, feature branches off mainline isolate custom development work, and releases branches off mainline isolate custom releases. A strict coordination of branch-to-branch merges is required to propagate changes between various combinations of branches without violating branch integrity.
Figure 1

Figure 1 – Traditional branch model for vendor-based custom 1.0 release

The diagram in figure 1 shows a branch model for a new project tracking vendor code and creating a custom release. The project is initialized by importing vendor code into the mainline3. In this scenario, the project is starting with the vendor 6.0 release. The vendor code is tracked independently on a dedicated branch off of the mainline, also known as the vendor branch (label a). Custom features are developed on dedicated feature branches, also off the mainline (label b). As feature development is completed on respective feature branches, they are individually merged into the mainline (label c). When the custom 1.0 release is scheduled, a custom release branch4 is created (label d). The custom release branch is used to prepare the release (eg. environment configurations), provide isolation for release specific customizations, and permit uninterrupted merging of future features onto mainline. Optional labels on the release branch can be used to identify release candidates or test builds. Once the custom release branch is fully tested, it is labeled as “1.0” and merged into the mainline (label e). The custom 1.0 release captures two custom features based on vendor 6.0 code independent of the unmodified vendor code isolated on the vendor branch.

The diagram in figure 2 is a continuation of the previous branch model highlighting a vendor upgrade, a patch fix, and a new custom release. Upgrading the vendor code requires importing and merging the new vendor release into the vendor branch, also known as a “code drop” (label f). In this scenario, the vendor code is upgraded to the vendor 6.1 release.

Figure 2

Figure 2 – Traditional branch model for vendor-based custom 2.0 release

Vendor merges are typically easier between adjacent vendor releases but still require careful attention especially when dealing with deleted files5 in the newer release. The vendor upgrade is conceptually treated like any other feature development with end goal of being merged into the mainline. A vendor-merge branch is created to merge between the upgraded vendor 6.1 code and mainline containing all current customizations (label g). The vendor-merge branch isolates6 the mainline from any merge-specific problems such as custom feature incompatibility or file namespace collisions7. If the merge is successful, the new vendor-merge branch is merged into the mainline supplying active development with the latest vendor upgrade (label h).  When the custom 2.0 release is scheduled, a new custom release branch is created (label m). In the meantime, a defect was patched on the custom 1.0 release resulting in a custom 1.1 release (label i). After releasing the custom 1.1 release, this patch is merged into both mainline and custom 2.0 release branch to prevent regressions (label k). After preparing the custom 2.0 release, all changes are then merged onto mainline (label n). The custom 2.0 release captures two (previous) custom features and a patch all based on the upgraded vendor 6.1 code.

One serious caveat with the above vendor-merge is that all custom features present in the mainline were merged with the vendor upgrade in the vendor-merge branch (label g). If only a subset of features were desired, the other features would have to be removed from the vendor-merge branch manually!  In fact, this is what needs to be done in order to preserve mainline as the central development codeline. Taking a step back, figure 3 shows a highly undesirable branch model for managing feature-by-feature vendor upgrade releases. To merge a subset of existing custom features with the vendor upgrade, the custom release branch would have to be based directly off of the vendor branch (label p), strictly isolated from mainline, and selectively merged with any original feature branches (label r). In this scenario, Feature-2 and Patch-1 are merged, but not Feature-1. But this violates the policy of mainline being the central development codeline and causes a decentralization of release branches!  Creating custom release branches off of both mainline and the vendor branch quickly turns into an unnecessarily complex web of branching and merging.

 

Figure 3

Figure 3 – Traditional branch model for managing feature-by-feature vendor upgrades

Relying on branch-based SCM models to manage customizations to vendor releases requires a complicated orchestration of merges between numerous branches. While this example highlighted only two custom releases, a single patch, and a single vendor upgrade, the situation quickly becomes overwhelming when considering multiple vendor upgrades and multiple custom releases. Lastly, additional overall complexity arises when considering the propagating of patches between mainline, compatible feature branches, vendor merge branches, custom release branches, or all of the above. By now it should be clear that the traditional branch based solution quickly becomes unwieldy.

How Streams Make It Easy

Imagine rotating the branch model in figure 2 clockwise 90 degrees, and allowing automatic inheritance of changes between adjacent branches. Now you are starting to think in streams8. AccuRev’s stream based SCM architecture intuitively models parallel development with independent, customizable software development workflows and implicitly makes merging simpler with automatic inheritance and a merge-early, merge-often paradigm.

Streams can be thought of as “intelligent” branches. A stream represents a specific configuration of source code. More specifically, a stream contains a specific version of each and every file visible to the stream. When a developer needs to modify code in AccuRev, they simply create a private workspace9 from any stream and instantly have writable access the all the files for that specific configuration. Streams can be organized into a parent-child hierarchy, also called a “stream hierarchy”. A stream hierarchy intuitively defines a promotion-based software development workflow. Each stream in the workflow represents a stage in the process such as development, integration, QA testing, or SOX auditing. Changes move from one stream to the next by being promoted. To control promotions, streams can be locked by user or role. For example, only members of the release engineer group can promote to or from the QA stream. Streams also have a unique, built-in feature that allows configurations to be automatically inherited from parent to child.  This inheritance allows newer file versions higher up in the hierarchy to become automatically available (for update) lower in the hierarchy. Imagine fixing a defect in your QA stream and having the patch automatically available to all 50 developer workspaces everywhere in the sub-hierarchy.

Figure 4

Figure 4 - Stream model overview for vendor-based custom releases

The diagram in figure 4 shows a stream hierarchy that models the parallel development of custom features based on vendor code. Vendor code is entirely managed in the base stream “Vendor”. The base stream serves as a repository for imported vendor releases. Snapshots off the base stream label vendor releases (label a) and serve as a named stable base10 for a version-specific custom development codelines11 (label b).  Each development codeline creates workflow streams named “Integration” 12 (Int), “QA” 13, and “Custom” to model the development process (label c). New development occurs in private workspaces (not shown) and is eventually merged, tested, and promoted through the Integration and QA streams (label c). Finally, the changes are promoted to the Custom stream (i.e. production), and a snapshot is created to label the official custom release (label d). The green box below a stream indicates that changes are present and inherited downstream. The lock icon signifies that promotions are controlled by user or group. Compared to the branch model, AccuRev’s stream based model more intuitively organizes vendor releases, custom releases, and custom development workflows.

 

Figure 5

Figure 5 - Stream model for vendor-based custom 1.0 release

Starting a new project based on vendor code requires simply importing the vendor code, creating a vendor release snapshot, and setting up a development workflow with streams. The diagram in figure 5 is a stream structure for a new project tracking vendor code and creating a custom release. The vendor 6.0 source code is imported into the base stream from a private workspace and a vendor release snapshot is created (label a). This snapshot serves as a stable basis for the 6.0-based custom development codeline. Features are developed in private workspaces and eventually promoted to respective feature streams (label b). Individual feature streams support collaborative development between team members. As features become mature, their changes are promoted to Integration to be merged with other features (label c).

Keep in mind that features can be promoted when they are either fully or partially complete. The frequency of promotion is directly proportional to continuous integration as promoted changes are immediately inherited by other developers down stream14. What is the advantage of automatic inheritance?  Inheritance makes merging easier because developers have the option to integrate with other (promoted) features before their own work is completed. Gone are the days of complicated “big-bang” merges at the end of feature development!  After testing in Integration, the changes are promoted to QA and subjected to black-box regression testing (label c). Optionally, release candidate snapshots off of QA can be used to capture test builds. When testing in QA is complete and the release is scheduled, the changes are promoted to “Custom” (i.e. production) and a snapshot is created capturing the custom 1.0 release (label d). So far, the stream hierarchy has organized vendor code independent of a custom release and provided an intuitive workflow for developing, testing, and releasing customizations.

 

Figure 6

Figure 6 - Stream model for vendor-based custom 2.0 release

The power of this stream model becomes further evident just after the first vendor upgrade. Figure 6 is a continuation of the previous stream structure highlighting a vendor upgrade, a new custom release, and a feature-by-feature merge. The upgraded vendor 6.1 source code is imported and merged into the base stream from a private workspace and a vendor release snapshot is created (label f). This snapshot serves as a stable basis for the 6.1-based custom development codeline. Meanwhile, in parallel, a custom patch to the 6.0-based codeline is developed, promoted, and captured in a custom 1.1 release (label i).

Now it’s time to consider migrating customizations to the 6.1-based development codeline. At this point, all tested and released 1.x customizations are located in the 6.0 “Custom” stream.  Which features should be migrated?  AccuRev supports promoting changes at the file level or the feature level15. Promoting by feature makes it very easy to pick and choose features without concerning over which specific files were involved in a given feature development – something very difficult to do with branch-based systems!  When the desired features are selected, they are migrated across codelines by promotion to a private workspace (label h). At the time of promotion, the migrated features will be merged16 with vendor 6.1 code (label k). Performing this merge in a private workspace prevents conflicts from being inherited by other developers.

When the merge is successful, everything is (eventually) promoted to Integration, QA, and 6.1 “Custom”. A custom 2.0 release snapshot is created capturing a feature and patch merge with the upgraded 6.1 vendor code.  After removing unused streams and workspaces, figure 7 shows the final stream structure modeling two vendor upgrades and three custom releases.

 

Figure 7

Figure 7 – Final stream model organizing two vendor upgrades and three custom releases
 

Conclusion

Managing third-party “vendor” based customizations requires an additional layer of configuration management to track vendor upgrades alongside custom releases. To be successful, vendor code must be tracked independently of the dedicated development codelines used to develop custom releases. The challenge is to integrate vendor upgrades with select customizations while preserving the integrity of the active development codelines. Branch based models utilize numerous branches and require a cumbersome orchestration of merging to be successful. AccuRev’s stream based model provides a more intuitive solution by using parallel codelines, stream inheritance, feature-by-feature merging, and a promotion-based workflow.  In general, as just described when developing vendor based customizations, the quality of software progresses from an immature state during development to a mature (tested) state in the release. AccuRev’s stream based model defines a methodical workflow that models the natural evolution of software.  Compared to traditional branches, the stream model presented in this article provides a more natural way to manage vendor based customizations.

 

Footnotes


1 Vendor merge: Not necessarily the most recent version of the vendor release.

2 Central codeline: Mainline is the source of all branches and the destination of all merges. This prevents creating a difficult to manage staircase model of branching where each subsequent release branch is based of the previous.

3 Vendor import: Some SCM systems have a built-in facility for handling vendor branches. Otherwise, adding the vendor code to a separate branch and merging into mainline is sufficient.

4 SCM Best practice: Release codeline isolates release preparation work and/or fixes from mainline development.

5 Vendor upgrade: By simply unpacking and overlaying the new release on top of the previous release will not delete files removed from the upgrade. To solve this, in a version controlled work area containing the previous vendor release, remove all files and unpack the new release. The SCM system will be able to compare the upgraded files with the previous release, already in source control, and detect new files (external), changed files (modified), and deleted files (missing). Once the files have been committed, don’t forget to apply a label!

6 Local-copy-merge:  Rather than merging vendor code directly into mainline, a branch is created to isolate the merge and prevent any conflicts from disturbing active development.

7 Namespace collision: The upgraded vendor code may have a new file with the same name as a customized file.

8 Additional information: http://www.accurev.com/documentation.html

9 Private Workspace:  Provides local versioning of changes independent of changes made by other developers. Sometimes referred to as private sandboxes, private workspaces allow developers to commit incomplete work, rollback to previous private versions, and serve as backup storage until work is complete.

10 SCM Best Practice: Work off a known, stable configuration rather than one that is constantly changing.

11 SCM Best Practice: Release lines isolate changes specific to releases from active development.

12 SCM Best Practice: Used for continuous integration of features and a controlled environment for smoke testing.

13 SCM Best Practice: Serves as a controlled environment for end-to-end, black box regression tests.

14 Inheritance Model: inheritance is calculated automatically but updating a workspace to retrieve the inherited changes is manual – AccuRev lets the developer decide when to incorporate external changes to their workspace.

15 AccuRev Change Packages: enabling change packages allows developers to contextually group changed files as a single “feature” set. Developers can be prompted to make the association when promoting file from their private workspace. In the development workflow, using change packages supports promoting or migrating “by feature” rather than file-by-file.,

16 Merging: trivial merges can be performed automatically while conflicts require manual intervention.

 

References

Berczuk, Stephen. Software Configuration Management Patterns. Addison-Wesley, 2004.

“CVS Vendor Branches”. http://ximbiot.com/cvs/manual/cvs-1.11.22/cvs_13.html. July 1 2006.

“Subversion Vendor Branches”. http://svnbook.red-bean.com/en/1.1/ch07s05.html. July 1 2006.

Perforce Vendor Branches”. http://www.perforce.com/perforce/technotes/note015.html. July 1 2006.

 

David Thomas is a Senior Sales Engineer at AccuRev, Inc.  Prior to joining AccuRev, David was a Senior Software Engineer at Orbitz.com where he successfully implemented AccuRev for an effective best-of-breed application lifecycle management (ALM) solution. Since 1996, David has been a programmer for many large Internet eCommerce Web sites where he has used numerous commercial and open source SCM tools.  He holds a B.S. in Computer Science and Mathematics from the Illinois Institute of Technology. You can reach David by email at dave@accurev.com .

Copyright © 2006-2008 AccuRev, Inc. AccuRev is a registered trademark of AccuRev, Inc.