Archive for May, 2008

Right Process, Wrong Tool? Getting Ready for Agile

May 30th, 2008

Yesterday I was a panelist for a Webinar on agile tools, focusing on software configuration management (SCM), build and software process automation (SPA), the latter term referring to the set of defined, repeatable and measurable automated development workflows that engineers use to transform requirements into shippable software products. Contrary to what I’ve read about the disdain that some agile devotees have for tools, most of the attendees were hungry to know what features their SCM tool should have in order to support agile software development and SPA. Here are some of the highlights, and of course, my take on why I think AccuRev is the best tool for agile software process automation.

There are five key feature areas that an SCM tool needs to support in order to be ready for agile:

* Support for flexible process models

* Continuous integration support

* Support for issue-based development

* Efficient branch and code management

* Private version controlled developer workspaces

Let’s take a look at each of these in turn.

* Support for flexible process models. Agile is often one of several processes being employed within a software development organization. Unless your SCM tool is flexible and process-neutral, you will have a hard time implementing agile (say, for product development) and more traditional processes like waterfall (for example, for product maintenance work) in the same SCM tool. AccuRev streams are a natural way to model any process, and thus are a good fit when agile needs to coexist with other development processes. As for software process automation (SPA), AccuRev streams again are a great fit, since they enable users to model any arbitrary stages of code transformation that a development team sees fit to define as part of their process. By adding triggers and workflow to a stream hierarchy, teams can implement SPA directly in AccuRrev.

* Continuous integration support. Continuous integration is one of the core process elements associated with agile development. By building and testing frequently and acting on the results of tests, teams can uncover defects or test gaps earlier in their development cycle, saving time and money compared to such discoveries late in the cycle. But continuous integration goes beyond just testing the nightly build. With multi-stage continuous integration in AccuRev, code is automatically promoted up the stream hierarchy into more stable configurations as it passes tests. At each stage, continuous integration takes over to build and test, typically with a wider scope of testing as the code nears the release stage. Legacy SCM tools make this type of automated integration factory somewhere between difficult and impossible due to the complexity involved in setting up the hierarchy and in automatically moving and merging code as it flows up the hierarchy.

* Support for issue-based development. Apparently there is a lot of contention about the need for filing issues and defects in agile development. This has puzzled me greatly. While I’m in favor of developers identifying and fixing issues as they are discovered, you lose valuable process information when a defect or enhancement ticket is not filed and later associated with a code change. Without an issue that describes what the problem was, someone looking at the code changes for audit purposes or for group code reviews is at a disadvantage. Why was this code change made? Is it related to other changes? How long did it take? Was it done to fix a bug or to add a feature. In AccuRev, issues either in the integrated AccuWork issue tracking system, or in a 3rd party issue tracking system, can easily be associated with code changes via the AccuRev Change Package mechanism. This establishes basic traceability between issues and the code changes that developers make in order to satsify the requirements of those issues. Issue-based development is well-defined, repeatable and measurable – all hallmarks of good software process automation.

* Efficient branch and code management. Any time you’re working on more than one project, you need to isolate that project’s code from other projects. With agile and multistage continuous integration, even a single project requires multiple code lines in order to separate in-progress code from unit tested code from system tested code that is ready for release. If an SCM tool makes branching, merging and labeling difficult, teams tend to practice branch avoidance, which I sometimes like to call “fear of branching.” This is a classic example of letting a tool dictate what processes you can implement. In AccuRev, streams replace branches as the mechanism for isolating codelines. Since streams are represented inside of AccuRev as data separate from the actual file versions, creating streams is fast – really fast, like, a second or two – and managing a system with hundreds of streams spanning multiple projects and processes is easy.  For continuous integration, AccuRev snapshots and time-based streams are also fast and easy to create and manage, and give users a straight-forward way to “label” an interim or milestone codeline without having to place markers in thousands of source files.

* Private version controlled developer workspaces. Software developers are the heartbeat of any engineering organization. Executives at any development shop will tell you that hiring talented engineers and keeping them well-tooled and productive is the single largest challenge that they face. For agile, this is even more of a challenge, since coding cycles tend to be shorter, and thus anything that gets in the way of individual or team productivity tends to have a greater negative impact on the project. Private version controlled workspaces like the AccuRev workspace model improve productivity, since they enable developers to work in isolation (while they are ‘heads down’ coding). Private workspaces in AccuRev also give developers full SCM capabilites in their workspaces without the need to share in-progress code prematurely. By using the ‘keep’ operation, developers make safe copies of their work in the AccuRev repository, and later can ‘promote’ the code to an integration stream to combine their work with that of their teammates. Individuals are more productive in this way, and if continuous integration builds are frequently testing the integration stream, so are teams.

In a nutshell, agile requires tools, and these tools need to support different modes of operation than with other processes. SCM can help or hurt you in setting up and executing an agile process, so these guidelines are a way to help you get your SCM tool ready for agile - easy of course if your tool is already AccuRev!

If you’re interested, you can view the webinar recording.

Is Your Software Development Environment Agile-Ready?
Free On-Demand Webinar

4 comments »

Posted in Best Practices, SCM Resources

Tags:

Take Back Control: Using LDAP for SCM Authentication

May 15th, 2008

Yesterday, just for fun, I counted the number of times that I logged into a computer or website. Once to login to my PC. Once more to connect to the company network. Three times for the 3 different UNIX boxes I needed to work with. And (if you promise not to tell my boss) once to login to a popular online shopping site to cancel a book order that apparently was lost in shipping. That’s six times in a day – and a slow day at that.

All this logging in got me thinking about security, authentication and of course, software configuration management (SCM) systems. Most SCM users, unless they are in the computer security business or are otherwise paranoid, don’t think about what goes on when they type in their user name and password and press Enter. In this post, we’ll peel back the covers a bit and show how you can use LDAP as the authentication mechanism for the AccuRev SCM system.

Starting with version 4.6, AccuRev introduced the notion of a ‘custom’ authentication mechanism. If you boil it all down, there are only three things that you need to do in order to use LDAP authentication with AccuRev:

1. Tell the AccuRev server that you want to use custom authentication

2. Create users in AccuRev and in LDAP

3. Write a special AccuRev trigger that authenticates the users against an LDAP server

Let’s look at each of these in turn. First, a word of caution. As with any change to a shared production system, it is best to practice this in a safe environment. If you don’t have a spare AccuRev server laying around, you can always download the free 30 day, 5-user evaluation kit and use it to fine tune your new authentication process.

Now to the details. To tell the AccuRev server that you want to bypass the built-in authentication mechanism and use a custom method, execute the following command:

accurev authmethod custom

Next, you’ll need to create some AccuRev users. In this example, we’ll assume that you’re already using LDAP for other applications, and therefore user entries already exist in the LDAP server. A typical user in an LDAP server might look like this in LDIF format:

dn: cn=James T. Kirk,o=engineering,dc=enterprise,dc=com
objectclass: top
objectclass: person
objectclass: inetOrgPerson
objectclass: organizationalPerson
cn: James T. Kirk
sn: jtkirk
mail: jtkirk@enterprise.com
userPassword: jtkirk

Well, typical if they happen to be the captain of the most famous starship ever! But I digress.

In AccuRev, we need to decide how this user will be represented. In this example, we’ll use the LDAP ‘commonName’ attribute, which is shown above as ‘cn’, as the AccuRev username. Here’s the command we’ll use to create that user in AccuRev:

accurev mkuser “James T. Kirk”

At this point, we have a user represented in LDAP, and that same user represented in AccuRev. All we need to do is to tell the AccuRev server how to authenticate this user via LDAP. We do this via an AccuRev trigger known as the ’server_auth_trig’. Here is some sample code for a server_auth_trig that does just that:

use Net::LDAP;
use Net::LDAP::Util qw(ldap_error_text);
use Net::LDAP::Constant qw(LDAP_SUCCESS
            LDAP_CONNECT_ERROR
         );

use XML::Simple;
use strict 'vars';

# Server info for contacting LDAP
my $LDAP_HOST = "localhost" ;
my $LDAP_PORT = "389" ;

# We explicitly list the subtree DN to use when rewriting the incoming username as an LDAP Bind DN.
my $ldap_baseDN = "o=engineering,dc=enterprise,dc=com" ;

# Default attribute name for binding.  This attribute is concatenated
# with the incoming username and the ldap_baseDN above to form
# a Bind DN.
my $ldap_bind_attribute = "cn" ;

sub main
{
    my ($xmlinput);
    my ($command, $ip );
    my ($username, $password );
    my ($result);

    # populate array using XML::Simple routine, reading from stdin
    $xmlinput = XMLin('-', forcearray => 1, suppressempty => '');

    # set variables
    $command = $$xmlinput{'command'}[0];
    $ip = $$xmlinput{'ip'}[0];
    $username = $$xmlinput{'username'}[0];
    $password = $$xmlinput{'password'}[0];

    # First, establish a connection to the LDAP server
    my $LDAP = Net::LDAP->new($LDAP_HOST, port => $LDAP_PORT) ;
    unless ($LDAP) {
                print "Unable to connect to LDAP server on host $LDAP_HOST at port $LDAP_PORT.\n" ;
                return LDAP_CONNECT_ERROR;
    }

    # Now that we are connected, rewrite the username as a DN and attempt to bind to the server
    my $ldap_bindDN = $ldap_bind_attribute . "=" .$username . "," . $ldap_baseDN;
    print "Attempting to bind as: $ldap_bindDN\n" ;

    my $mesg = $LDAP->bind($ldap_bindDN, password => $password) ;

    # 'code' method contains any error code from the bind call
    # including success, so we return it to the caller
    my $return_code = $mesg->code;
    print "LDAP_BIND returned: $return_code\n";

    # Now unbind to free the connection
    $LDAP->unbind;

    # return the auth code to the AccuRev server
    exit ($return_code);

}

# run main routine
&main();

The main trick is to ‘rewrite’ the incoming user name in the form of an LDAP Distinguished Name, or DN, and then to use that DN and the incoming password to bind to the LDAP server. Binding is a fancy word for logging into the server. Typically this is done by providing a DN (to uniquely identify the user) and credentials (in this case, a password).

As we said earlier, we’re using a convention in this example that the incoming user name represents the ‘commonName’, or cn, attribute of the user’s LDAP entry. We then construct a string by concatenating the cn with a hard-coded base DN, the latter representing the subtree within the LDAP server where the users exist. The resulting DN in this example is:

cn=James T. Kirk,o=engineering,dc=enterprise,dc=com

which is represented in the example as the perl variable $ldap_bindDN. If the bind is successful, the trigger returns a 0, and the user is logged into AccuRev. If the bind fails, the trigger returns a non-zero code, and the user login is rejected.

There you have it. A few simple steps and you can use the industry standard LDAP mechanism to provide authentication for your AccuRev users. LDAP is used in all sorts of enterprises, from education to technology companies to government, and so is AccuRev, so we’re glad to provide a way for our customers to use this powerful and ubiquitous authentication mechanism.

4 comments »

Posted in AccuRev, Tips and Tricks

Tags:

Continuous Integration: Methods of getting change

May 14th, 2008

Do you remember the last time you were excited to go somewhere? Were you like a kid saying, “Are we there yet? Are we there yet?”

More likely you were the one getting ‘them’ there. I’m sure it got pretty annoying to keep hearing everyone in the car moaning, wanting some sort of distraction until they got there. Maybe you even had DVD players or some other distraction so you didn’t have to hear the questions.

Now think about how you use Continuous Integration (CI) . Do you have polling of your software configuration management (SCM) tool setup? What about your other process tools, do they poll your SCM as well? Guess what, it’s the same difficult trip. You have tools burdening your working SCM (who slaves away everyday to provide for these other tools!). Every hour/minute/seconds some tool is asking your SCM tool “Are we there yet?”. Doesn’t make a lot of sense does it? Think about how many tools you have running daily. You might have multiple CI machines setup, multiple reporting machines, deployment machines all asking the same question over and over.

Quite a burden.

Now think about what your users are doing during this time. They are looking for the same distractions you might have given the kids. They are off emailing their buddies about the latest game, or watching AccuRev on Youtube (OK, maybe something even more enjoyable on YouTube). Doesn’t sound very productive, yet these automation tools were meant to do just that, increase productivity.

So what do you do? Well, a lot of tools allow you to flip the model. Push your information, don’t pull it.

If you have a large enough development group this won’t be enough. Maybe there really are code changes going into the system every minute. If you also increase the granularity of the information going to your CI and reporting tools, these tools can then decide the correct time to be a burden.

You can also reduce the frequency that you give out the same information. If you have several tools (or stages in a tool) depending on the same answer, they can get the answer from a secondary source that gets populated once.

You also want to be diligent about your monitoring. If you see a periodic load on your tools, justify the load. If it looks strange that Kevin is checking the history of a stream every minute, it probably is. If instead you saw CIMonitor as the user it would be more explicit. And it would be obvious that this should change.

And really, changes like these apply to any tool you are using. Do you really need updated reports everytime a bug is fixed? What about every day instead? If you reduce the burden on your tools to a ‘necessary’ level, then they can be further used to answer other questions.

Are we there yet?

No comments »

Posted in Best Practices

Tags:

Why Use AccuRev for Document Management?

May 8th, 2008

Let’s face it, process-centric SCM is not just about the process, code and issue tracking, it’s also about document management. Simply storing documents in a shared network space is no longer sufficient; documents related to a product release need to be versioned and tracked. We’re talking about a wide range of documents here — UML diagrams, meeting notes, functional specifications, wireframe prototypes, product plans, slide decks, planning spreadsheets, and, of course, end-user documentation from release notes to complete online help systems.

AccuRev’s stream-based, process-centric approach makes it easy to manage your product-related documents right alongside your code and issues. Add a couple of folders to your workspace — call them planning, doc, or whatever you like. Add your planning documents to the folders, and promote everything. Now your product documents are part of the stream.

  • When you create a child stream for the next release, those planning documents are included automatically because of AccuRev’s built-in inheritance. Just update the documents for the new release and promote. Product managers and others can create their own workspaces where they update those documents as needed.
  • Want to track general-purpose planning documents that aren’t related to a specific release? Create your child stream under the root, and store all those documents there. Create child streams and workspaces based on document evolution, not product releases.
  • Want to free your product managers and technical writers from using the AccuRev client interface? Now there’s the AccuBridge for Windows Explorer (for AccuRev V4.5.4 and newer), which provides AccuRev functionality from a right-click menu in Windows Explorer, and icon decorations to indicate AccuRev status.

Documents represent knowledge; they are part of the company’s history. They provide records of decisions and the context in which those decisions were made. They show the reasons behind the implementation decisions, the backtracking, the roads not taken, and the plans for the path ahead. When these important documents are not versioned and tracked, information is lost.

1 comment »

Posted in AccuRev

Tags:

Globally Distributed Developers, Under a Single Roof

May 7th, 2008

One of the most common and problematic challenges that exists in today’s software development environments is how best to support a Globally Distributed Development organization. In ye olden days, you had the entire team co-located in the proverbial cube farm under a single monolithic roof. If Brad wanted you to review the code he just wrote, he would literally turn around in his chair, ask you to come in and look over his shoulder.

Times have definitely changed. Now, your team might be headquartered in Boston, separate R&D sites in California and London, with some specialized groups in Bangalore and Shanghai. But that’s not necessarily the hard part. Where it gets complicated is when all these developers are trying to work on the same source code at the same time. Mastership issues questions, latency, mismatched process across sites; communication problems, lack of project visibility; these things all lead to significant decrease in productivity, not to mention the chaos for those trying to manage the effort.

Enter AccuRev. Uniquely architected to support remote and geographically distributed development (GDD), there are several key built-in capabilities that make the challenges of the past disappear. Consider the following graphic:

  • AccuRev’s Stream Browser presents a dynamic visual representation of the software development process that is both fundamentally tied to the source code itself as well as being flexible and enforceable. At a single glance, *anyone* working on the project anywhere knows exactly what the process in place is. (See example AccuRev annotated screen shot here from the Alaska Airlines success story).
  • Geography has zero impact on your position in the process; a developer in the UK can happily use a Workspace on the same parent stream as a developer in the United States. For low bandwidth locations, AccuReplica can provide LAN-quality access without introducing the traditional mastership and latency problems of other replication solutions.
  • The private nature of the Workspace means that these remote developers can “share” code while still determining when to deliver their changes publicly.

Here’s the scenario: Developer ibergman works out of London, while jtalbott works in Boston. However, they are both part of a virtual team working on ComponentC. With AccuRev, the normal boundaries and limitations of time and space – not to mention being constrained by an inadequate SCM tool – no longer apply. Okay, I took some verbal liberties with the “time and space” bit, but it’s actually not too far from the truth.

In London at noon, ibergman wraps up a section of code she’s been working on and performs a Keep, which in AccuRev is a private check-in. The change is versioned yet stays within the confines of the Workspace, not yet ready for public consumption. But ibergman wants some validation, and asks jtalbott to review her code. Using instant messaging (IM), she pings him and catches him as he’s having his first sip of coffee, probably a colombian supremo. In other tools, how would someone review a private change that was just committed on the other side of the ocean? Would they even have private commits in the first place? In AccuRev, the moment ibergman performed that Keep in London, a visual identifier is available to anyone viewing the StreamBrowser, such as to jtalbott in Boston. So jtalbott clicks on the icon, and now has immediate access to operations like View, to see the file, and Diff, to compare against any previous revision of this file:

Did I mention that jtalbott’s access to these operations is instantaneous, as soon as ibergman performs the Keep? He can even take her version and send it into his own workspace if he finds it interesting enough to want to do additional development on.

So the previously mentioned problems of mastership, latency, visibility, communication, and most importantly Process, have all gone away. No more waiting on a 24-hour turnaround to get that Shanghai code copied into the branch. No more working in the dark not knowing exactly where your piece of code fits into the puzzle. Each team can regain the responsibility of merging in their efforts into common integration points, using a well-defined process implemented with streams.

It’s a remarkably simple and elegant solution to a complex and challenging problem. Of course, it’s still not going to solve the amusing problem of both the London and Boston developers feeling like they are superior to each other, but at least now they can actually review each other’s code real-time to help figure that one out ;-)

No comments »

Posted in AccuRev, Agile, SCM Resources

Tags:

The Quest for Successful Parallel Development

May 1st, 2008

Read my earlier post: Fortune 500 Software Company Answers Your Questions on Using AccuRev

Up until fairly recently I had never worked anywhere that was able to do parallel development without it becoming a nightmare. Individual contributor or manager, we just seemed unable to find that secret magical incantation that would lead to a truly successful parallel development cycle.

Every time the team had a different plan, and the project’s champion would trumpet why this time ‘things would be different’. But in the end, we always failed in one of two areas (sometimes both):

  • The quality of the software suffered dramatically
  • We lost control of the schedule and we were forced to ship late

I actually worked on one project where the resulting release was so bad that a few weeks after release we had to pull the product off the shelves and work frantically to repair the damage.

Then a few years ago my team was looking for a new software configuration management tool to use. We started out wanting ClearCase, but after much deliberation, decided on Accurev. At first, we didn’t realize the power of the tool now in our control. But after a release or two, we started to realize what we had and adjusted our process accordingly and began to achieve very successful, high quality, and on time releases done in parallel with one another. We had found the holy grail of project management!

We started to understand not just what the best practices were with Accurev, but to understand what we were getting by following them. We made other changes to our processes that helped as well, but I give a lot of the credit to the tool-set for helping us in the right direction.

We started out performing a round-table series on our past projects, looking for common themes, and then choosing the top themes and seeing how we could use Accurev to solve them, and here are some of the things we found:

Problem: Often parallel projects require extensive code changes to the same files, leading to a need for developers to communicate what is being done to which files and when. Often this communication is not handled well.

Solution: Accurev’s work in progress and annotate commands, as well as the version browser allow developers to quickly view who else is making changes to the files that they care about, and what is being done to them. Now developers can pro-actively gather the information they need instead of depending on someone else remembering to tell them.

Problem: Often projects have late phase merge points which are both complicated and error prone. In addition since merging is generally done in the mainline, the entire release can be held when any two projects try to merge together.

Solution: By allowing stream definitions to be dynamic, Accurev allows developers to create common merge streams ad hoc at no cost, and to join the projects without effecting any other users. It also provides excellent trivial merge capabilities that save the users much of the error prone work, calling out only the non-obvious decision points which greatly simplifies the merge task.

Problem: Often bug fixes get made to one ongoing project, but developers forget to make the same fixes to other active lines of development.

Solution: Using change packages users can quickly make changes, move them to all appropriate locations, and quickly verify that the tasks were completed successfully. Using the stream inheritance feature minimizes the number of lines where a developer must make a fix for his coworkers to take advantage of it. This greatly lightens the load on developers and cuts down on the chance that something will be forgotten.

Problem: Ensuring 100% reproducibility of a build from pristine sources is a huge challenge. Changes to the build system environment, tools and code changes can all effect your ability to make a minor fix to released software.

Solution: By keeping your complete build system (environment, tools, sources) all in Accurev, and by using a snapshot you are able to guarantee 100% that what you try to build today is the same as what your built the day the snapshot was taken. Unlike a label, a snapshot is forever! This enables you to reduce the regression test matrix and focus on only the changes that developers make to the released software.

While I am sure that other SCM tools can offer similar results if you know where you are starting and know where you want to go. But what I marvel at is that using Accurev really lead us to the cup so that we could drink from it and enjoy the benefits of successful parallel development.

Have you seen the AccuRev 2-minute demo?

1 comment »

Posted in AccuRev, Best Practices, Customer Guest Blogs, SCM Resources

Tags: