Continuous Integration: What’s Not to Love?

July 29th, 2010 by clucca No comments »

It seems like everyone loves continuous integration.  I’ll come out and say it- I love continuous integration! When we talk about the most widely adopted Agile practices, this one comes up the most.  Its positive benefits as a feedback mechanism provide a quantum leap forward in how development organizations think about their code.  I find it very difficult to see any downside in doing continuous integration, seriously what’s not to love?

Modern day continuous integration servers have 3 functions: Detect if a new build is needed, execute build, and notify people of the results. This is a great way of facilitating feedback to developers and allowing them to adapt and resolve problems.

But there’s something lurking in the shadows that nobody is talking about, maybe because people aren’t even aware that it’s a problem. Maybe it’s because we don’t want to ruin our love affair with CI and we’re all in denial.

What happens when those builds are done? What about the rest of “it”?

When I say “it” what I really mean is:

Most development environments include such things as complex application servers, automatic testing, release processes, compliance, audits, databases, 3rd party libraries, build dependencies, code analysis, unit tests… and another 1,000 other things I don’t have enough space to list here. How can you deal with this? Getting feedback to my dev team is a great targeted way to let people know code is broken, but isn’t this feedback useless if you can’t get the product out the door?

If I take an example build lifecycle of an application which is:

1.)    Build Application

2.)    Test

3.)    Deploy to environments (DB) (APP SERVER) <- by hand

4.)    Test

5.)    Redeploy (DB) (APP SERVER) (PROD) <-by hand

This may seem like easy in this example, but if we took all of these steps in the real world, this could represent hundreds of servers.  And this process will have to be repeated for every version of the product.

The crux of this issue is that if any of these steps are not performed 100% correctly, it translates to real dollar$ lost for the organization. These operations have to perform like clockwork.

Taking Continuous Integration to the Next Level

This is why I believe bringing continuous integration to the next level starts with the concept that the build produced from the CI server is just the beginning. Setting up a simple CI server and producing a build is easy, but managing it through the rest of its life is the real trick.

In a real example of this, we could take

1.)    Build

2.)    Run automated tests

  • If test succeeds: Deploy to to environments (DB) (APP SERVER)
  • If test fails: Notify dev team

3.)    Manual Testing in QA environment

4.)    Approval process

5.)    Redeploy (DB) (APP SERVER) (PROD)

Using AgileCycle RM

In this scenario I’m taking a version of the results from a CI build, run automated tests on them, monitor the test output and wait for success or failure. If there is success then deploy all components of the application, this includes a database components and a java war component. The build will then sit in QA for manual testing until its marked approved by the QA team, managers and operations team for deployment to production.

The idea here is if we automate these processes and decisions based on build, automated tests and approval process, you can produce code quickly and at a more rapid pace. If you can produce a clockwork-like automation around your build/test/deploy related processes, your team can spend time on what’s most important: Getting code out the door.

AccuRev’s New Web Interface 2010.3

July 27th, 2010 by AccuRev No comments »
AccuRev’s Web Interface 2010.3 has just been released.  Web Interface 2010.3 is a Web application that runs on the Apache Tomcat server that allows users to access data managed by the AccuRev Server via a browser.

Highlights of Web Interface 2010.3 Release

The release of Web Interface 2010.3 includes several enhancements and bug fixes, including:

The ability to retain font size changes in Diff displays between sessions, and the ability to specify column widths within an Issues table.

The ability to Diff the results of two queries, and the ability to compare the results of an Issue Diff against the results of a query.

Better handling of redundant login attempts when the network is slow.

The ability to print AccuWork charts, and the ability to export to xml or to print all tables.

A new URL option to display the history of a specified transaction.

An upgraded on-line Help system.

The ability to Promote from the File Diff tab.

The ability to Send to Issue (specifying basis) from the Version Browser context menu.

The ability to perform Diff from multiple file selections in the Changes tab.

The ability to Diff against other streams (not just the backing stream) from the Stream Hierarchy pane.

Many Stream Browser enhancements.

Smarter context menu behavior, and better default behavior for the the Query Editor “is in” and “is not in” conditions.

The ability to optionally include the display of incomplete issues in the Show Active Issues view.

Greater availability of toolbars and more toolbar options throughout the Web Interface.

AccuRev Web Interface 2010.3 is available on the main AccuRev download page: http://www.accurev.com/download.htm.

AccuRev’s Agile Methodology Workshop

July 20th, 2010 by BDeMaria No comments »

AccuRev hosts educational Agile methodology seminars called “Agile Comes to You,” which reach audiences nationwide and focus on teaching best practices of Agile software development.  The seminars have been quite successful, and regardless of their organization’s level of Agile adoption, I know attendees have learned some great information from these sessions.

AccuRev doesn’t host the Agile methodology seminars alone, and generally presents in conjunction with partners Rally Software, Urbancode (the makers of AnthillPro), and Coverity. The seminars consist of a keynote with extensive Agile experience, educational presentations, and a short tools demonstration. The format has been so successful that we have used it for over a year, and you might even notice similarly-formatted seminars from other organizations as well. (Imitation is the most sincere form of flattery, right?)

The Agile Methodology Workshop

We try to focus on making our seminars as educational and relevant as possible by giving attendees access to the real life Agile experiences that presenters bring to the table.  So in addition to presentations focused on benefits of the Agile methodology and best practices, we came up with the concept of an “Agile Workshop.”

The Agile Workshop  allows each attendee to discuss their most difficult challenge in transitioning to Agile with other attendees in small groups, as well as with our Agile experts.  We do this for two reasons:

1) It gives the attendees a chance to exchange thoughts and solutions regarding their Agile migration.

2) It allows the attendees to interact with the panel of experts on how to solve these difficult challenges.untitled AccuRevs Agile Methodology Workshop

Once the group has discussed the challenges each individual faced during a transition to Agile, they then agree upon a top challenge that they ask the panel of Agile experts to comment on and offer advice.

For example, at a recent seminar in Toronto, this was the attendees list of top challenges:

  • Culture Change / Rest of the Organization not Agile
  • Support Agile and Traditional projects in parallel (Hybrid Process)
  • Massive/Distributed applications implementing Agile
  • Propagating user stories across multiple release lines
  • Agile with Distributed Teams
  • Agile with Outsourcing
  • We have been seeing this same pattern across most of our seminars, and I believe it gives us good insight into the state of Agile adoption.  It is amazing to see that even across very different organizations, the challenges that arise with Agile adoption are remarkably consistent from seminar to seminar.  It seems that no matter who you are, or what stage of Agile adoption you are in, many are facing the same challenges when moving towards Agile development. There is some comfort in numbers, knowing that you are not alone in facing hurdles.

    While I won’t take the time to answer every one of these challenges here today, I plan on commenting on each one of these issues in the coming months, in hopes that sharing my experiences and alternatives help you in solving these difficult problems.  I would also like to invite some of our Agile experts, as well as our attendees, that are internal to AccuRev or our partners to comment or blog on some of these topics to share some of their experiences.

    While the “Agile Comes to You” tour is taking a short break for the summer months, be sure to look for us in your city this September or stop by and visit us at Agile 2010 Conference in Orlando.  Have a great summer!

    Yes, You Can! Doing Agile with Remote Teams

    July 15th, 2010 by clucca 5 comments »

    This past year I’ve attended several Agile conferences, presented at many of our own conferences, and traveled to Agile tradeshows sponsored by some influential industry-leading names. What surprises me most is the variance I see on the answer to this question: How do I do Agile with remote teams?

    Some of the pure “Agileistas” will may answer this question in a manner that isn’t very possible for some of us in the real world, with “Don’t do it” or, “Reorg your company.”

    I don’t’ know what those people expect here- is it possible that you can convince your management organization to tear down its office walls, move entire teams from across the country into one office space, just because you heard it at a conference that it was going to be really hard to do Agile remotely?

    I certainly don’t believe that doing Agile with remote teams is a bad practice, nor do I believe that it’s impossible. Challenging? Yes it is. Easy to mess up? You bet. But there are some simple things you can do to avoid some of the pitfalls of remote organizations.

    Agile with Remote Teams

    Use face to face communication methods:

    I just got the iPhone 4. It has face to face video chat. I also use Google Talk, and this also has video chat built in. It works great! With all the communication technologies we have now a days, there is no reason to avoid personal contact with remote teams.

    If the remote team is a faceless organization, it will become a perceived impediment for the local team if there are problems. They wont’ be treated like part of the team, but more like an outside entity that drops code in and risks messing up the release.  We can bring these teams closer together to encourage communication, and allow them to adapt and respond to each other as issues arrive. Creating a persona and human link turns those faceless “code drops” into real people, people who you can reach out to. This gives the team the power to self manage your priorities, impediments and conflicts.

    Create Agile ambassadors

    We can even take face-to-face chat on the internet up a level. Sending ambassadors back and forth from the remote teams to home base and vice versa creates a human link that is deeper than any piece of technology canAgile for Remote Teams provide.  The ambassador’s job is to strengthen this link, because if the link is strong, each side will be more inclined to help each other.

    Sometimes having a planning session with the remote team doesn’t give them the overall sense of how important the stories you’re working on might be. They may not feel as if it’s important, and that’s because they don’t know all the juicy details that led up to the creation of that story. Having an ambassador at that site gives that team visibility into all of the bits of information that make one user story important. In other words, the ambassador gives the entire back-story to an iteration (IE the gossip) so they can get a sense of how important something is, it’s not just a priority number in the ITS.

    Use Tools That Work Globally

    With all of the face-time, ambassadors, and communication, it’s essential that teams have a global view of what’s happening during the development cycle. It wouldn’t make much sense to reach out and then not provide a way to extend that communication on the development level.

    Agile for Remote Teams

    Imagine a team where having access to a user story or a piece of code wasn’t easy and available to them? This handcuffs the team tremendously.

    Any remote team will need to be able to:

    • See each others user stories and tasks
    • Enter updates to user stories and tasks
    • Diff baselines and branches
    • Check out code from remote teams
    • Contribute to team discussions and wikis
    • Run continuous integration globally

    Use Multi Stage Continuous Integration

    Using multistage continuous integration lets people take a look at what’s been built, and if it functions correctly, give it to the other team. Having multi-stage set up gives you a way to integrate early and often, but only deliver changes that are “done”.

    One of the main problems with remote development is integration, and it’s a double edge sword for most SCM tools. If you isolate the remote team too much, they won’t integrate often. And when they do integrate to mainline, they may break functionality. The problem with this is that they will not be able to respond to that change for 6-12 hours if your team is in another country. This basically means downtime for everyone.

    But with multistage CI and AccuRev you can keep that team isolated and integrated at the same time.

    Is it possible to do offshore Agile?

    I’m not sure if it’s a question if it’s possible, I don’t think we have a choice. Offshore development is a reality that isn’t going away, and the simple answer of bringing teams together to practice Agile isn’t always variable.  Doing Agile with remote teams isn’t’ a choice, it’s a reality.

    Continuous Integration- Like a Perfectly Functioning Toaster Oven

    July 12th, 2010 by clucca No comments »

    Where would we be in the world if we didn’t have feedback loops? Whether you know it or not, feedback loops are everywhere and you probably use them every day. Your toaster oven, your car, and the computer you’re using right now to read this blog post- feedback loops are everywhere.  Heck, even most biological organisms including humans and animals are constructed out of feedback loops. (Don’t believe me?) It’s the basis of all life!

    A classic example of a feedback loop is the temperature control on your toaster oven. Toaster 300x300 Continuous Integration  Like a Perfectly Functioning Toaster Oven

    Let’s imagine that we set the temperature to 300 degrees.

    Here is the loop in which your toaster oven sets its temperature to 300 degrees:

    1. Check thermostat for temperature

    2. If greater than or equal to 300 degrees, stop heating coils

    3. If less than 300 degrees, heat coils

    4.  Goto step #1

    “So what?” You’re probably saying to yourself, that’s easy.  Provide feedback to a device and it’s behavior will change. Why is this a big deal?

    Continuous integration is the feedback loop for your development organization. It’s a feedback loop for people. It allows your developers to rapidly change course at a moment’s notice.

    The Continuous Integration Feedback Loop

    Let’s run through the continuous integration feedback loop, which is remarkably similar to the toaster oven.

    Here is the feedback loop for a developer while using continuous integration:

    1. Checkin code

    2. Receive e-mail of build results

    3. If build/test is success, enjoy the fruits of your labor

    4.  If build/test is broken, fix and Goto step #1

    If you don’t believe a feedback loop is necessary, if you’re thinking everything is working fine, you’re fooling yourself!  The reason? If there is a problem, you’re just not aware of it. In the same sense, would you just turn off the thermostat in your toaster?

    Here’s how the toaster works without the feedback loop:

    1. Heat coils

    2.  Burn down house

    And that’s what NOT doing continuous integration is like… playing with fire.

    A lot of times I hear from customers who aren’t doing continuous integration that it’s just too hard to implement CI in their organization. The reasons vary, but usually come down to not being able to afford rewriting manual processes already in place in order to gain a rapid, consistent build running. To this I say, you are using the wrong tools, and bad practices to boot. The amount of rework and frustration you will save will be worth your effort. It’s a fools game not to expose your integration problems to the rest of the organization.

    When I talk to customers who HAVE implemented a CI solution, I’m astonished at what I hear. They say “We never know we had so many issues until the QA cycle” or “We can’t’ believe we ever lived without this.”

    Changing course is one of the key principles of Agile; inspecting and adapting is what allows your team to avoid disasters. It gives visibility into what’s happening with your current processes. Continuous integration is the tool that you need to allow your teams to function like a perfectly functioning toaster oven.

    So what does a world look like without feedback? Houses on fire, cars crashing, development organizations failing and developers blindly making disastrous coding changes. Do yourself a favor, and think about feedback.

    Clearcase Multisite Unsynced

    June 30th, 2010 by AccuRev No comments »

    Here is one more AccuRev vs. ClearCase video to share.  And this one is a crowd favorite.

    AccuRev vs. ClearCase in ClearCase Multisite Unsynced

    Sure, we all know geographically distributed development teams face lots of challenges.  But ClearCase doesn’t provide a simple solution to this problem.  In fact, with ClearCase Multisite Unsynced, teams have trouble syncing up and don’t allow developers to work the the same branches at the same time.

    With AccuReplica from AccuRev, all remote teams can work together and on the same code, like one co-located development organization.

    Breaking Down User Stories

    June 28th, 2010 by damonpoole 1 comment »

    There’s a lot of excitement about getting to shorter iterations. But with a shorter iteration comes a higher degree of difficulty. One of the problems with shorter iterations is getting all of your user stories to fit into them. For many stories it is no problem, but then there are the stories that seem to only be doable in the time it would take to do two or three iterations. What can be done? Is it even possible to fit all stories into a short iteration?

    Before we get too far, consider this. Prior to Agile, what was the incentive to break a large task down into smaller chunks? If you thought it might take 2 months, why not take two months? After all, the release isn’t for 6 months. So, prior to Agile, even if something could be broken down, there wasn’t really any reason to do so. As a result, we’ve had few reasons to build up our bag of tricks for breaking tasks down. Rest assured, there are many techniques for breaking big user stories down into smaller user stories and in this post I’ll show you one of those techniques.

    Fitting User Stories into a Short Iteration

    New Picture1 300x196 Breaking Down User Stories

    The diagram above represents a story that has a certain amount of value. In order to get that value, we have a bunch of work to do. That work cuts across the front end, the middle layer and the back end. An obvious way to split up the work is to do it a layer at a time.

    But a user story is only a user story if it provides value to the user. Splitting a story up into “As a user I want the backend for X,” “as a user I want the middle layer for X,” and “As a user I want all of the UI for X” does not provide value until all three stories are complete. Stories need to provide value on their own.

    New Picture 11 300x176 Breaking Down User Stories

    Instead of breaking a story up into layers, try splitting it up by value. In this diagram, the original story has been broken up into three separate stories, each of which holds value in and of themselves. As a result, each story requires less work to implement. This process can be repeated (if desired) until you can no longer derive stories that still provide user value. At that point, each story is already as small as you currently know how to make it.

    A typical example of breaking down a user story is “As a user, I want a calculator function.” This might be for a mortgage application, where the user wants a handy calculator that they can use as part of the application process. One might be inclined to provide a full-function calculator right out of the box. But it may turn out that all the user really needs is to be able to cut and paste numbers out of mortgage forms, into the calculator, add them together or subtract them, and then paste them back into the form.

    If you are concerned that the calculator won’t have enough functionality at release, consider that (at least in this contrived example) they can always do what they want another way. But for some subset of users, you will have provided just what they needed. Based on user feedback you can then provide the next level of functionality. In just two short releases you will have provided more of what the end users really need than if you had predicted usage and done just one larger release.

    You don’t need to split stories indefinitely, you only need to split them until they fit into your desired iteration length. As you split more and more stories, you will find that it gets easier and easier, and you will start to think in terms of smaller units of deliverable user value.

    More Issues with ClearCase?

    June 25th, 2010 by AccuRev No comments »

    Due to positive feedback from our first blog promoting AccuRev vs. ClearCase parody videos (see the post Issues with ClearCase?), and seeing as ClearCase has not addressed the needs of modern software developers in years, it seems only fitting to share a few more.

    Rationally Wrapped

    While ClearCase users deal with complicated wrappers and scripts during upgrades, AccuRev is easy to use right out of the box, and doesn’t need any wrappers or scripts.

    Developers Revolt

    Locked out of ClearCase again?

    AccuRev will never lock developers out.

    Do these issues resonate with you and your development team?  Read more about how AccuRev auto-synchronizes with ClearCase, providing developers the ability to get up and running with AccuRev at no risk!

    Change Packages for Agile Workflow

    June 24th, 2010 by damonpoole No comments »

    User Story Based Engineering

    In other blog posts I’ve talked about using Multi-stage Continuous Integration to scale Continuous Integration for use in multi-team environments and using streams or branches to model your workflow directly in your SCM tool. While both of these approaches work well and provide a lot of value, they can also present a new challenge: how to associate enhancement requests and defects, which can be lumped together using the term “work items,” with the changes that implement them?

    If you are doing mainline development, life is pretty simple. On check-in enter the enhancement request ID or defect ID in the comments. Once that association is made, it is fairly straightforward to figure out which enhancements and defects have been checked in. On the other hand, the more people you have checking in to the same branch, the less likely it is to be stable, even if you are doing Continuous Integration. So, how can we maximize the benefits of using streams to represent process and integration stages and still have the ability to match work items to the work done to implement them?

    Representing Process and Integration Stages with Change Packages

    (Let me first point out that while I’m using the generic term “work item” here, in an Agile team one would most likely use the term “User Story.” Both apply equally well in this context, but I’ll stick with work item for now.)

    There is an SCM concept that maps to a work item which can be tracked from place to place, and that is a “change package.” It is called a change package because it represents something which, once defined, can be applied to multiple places and tracked regardless of how many different places it has been propagated.

    Change packages can be promoted individually or in groups from stream to stream. That way, as work is promoted from stream to stream you have a record of the content of each stream from a high level perspective instead of a files perspective. A list of files is not very meaningful or useful. On the other hand, change packages allow you to ask questions like “which user stories are in the stream that represents all user stories that are ‘done’,” without having to resort to mainline development.

    Something’s A-Twitter in My Backing Stream

    June 22nd, 2010 by amonty 3 comments »

    One of the cornerstones of any successful organization is communication. On Agile teams, we often meet to share information, update one another on progress, to reflect on that progress and discuss how the process can be improved. These interactions, be they stand-up or retrospective meetings, provide this information at regular intervals. But what if you need “real-time”  information sharing?

    Consider the concept of “one piece flow”, often treated as the holy grail of engineering process purists. Lean/Kanban fan boys live to talk about this at conferences, and in the isle outside my cube. The idea is that a single “workpiece” at a time moves through the workflow. In the software development process, this is rarely limited to a single item, but instead is throttled by WIP limits to minimize the bottlenecks in the process. For example, the number of items worked on by developers is limited so that the count of items designated “ready for test” is kept at a manageable level. As a tester on an agile team, I’d like to know as soon as an item is moved from the “wip/development” phase to the “test/validation” phase.

    Consider the following basic stream structure -

    Simple Stream Structure

    In an ideal model, testers would be notified as soon as an issue is promoted from the WIP stream to the Validation stream. This model assumes that the project is utilizing AccuRev change packages to track work items as issues. Having a distinct stream for validation purposes simplifies the job of the tester. All issues that exist in this stream have been unit tested by developers, passed basic regression tests, and are ready to be explored, validated, and promoted to the next stage in the workflow.

    So, how can this level of communication be achieved? AccuRev triggers! The rest of this post will demonstrate how the server post-promote trigger could be used to provide updates using arguably the king of “What’s Happening?” – Twitter.

    Why use Twitter in your Backing Stream?

    Why not? Sure, you could use e-mail. But who wants more e-mail? Tweet it, and let them aggregate the information for you. Team members can follow the account, or not. Pointy-haired managers that dream of pie charts and love visibility can subscribe to an RSS feed, that you don’t have to manage. Lastly, if you’re an agilista, you’re already hip and trendy, (and let’s face it, you probably  already tweeted about the awesome presentation on Lean that you saw at the latest Agile conference).

    The Server Post-Promote Trigger

    I am not going to go into the gory details of the AccuRev trigger system. Here’s what you need to know.

    1. The trigger can be written in whatever language you like. In this example, I use Ruby.
    2. The trigger needs to be registered with the AccuRev server, and associated with the depot for your software project.

    For this exercise, I wrote a quick script called tweet_post_promote.rb. The first step is to register this script with AccuRev.

    >accurev mktrig -p SoftwareProject server-post-promote-trig ~/dev/ruby/tweet_post_promote.rb

    Once registered, the trigger will now fire for all promotes in the SoftwareProject depot. Let’s take a look at the contents of the script.

    #!/usr/bin/env ruby

    require ‘rubygems’

    require ‘crack’

    require ‘twitter’

    ACCUREV_DIR = “/sandbox/amonty/accurev/”

    ACCUREV_STORAGE = “#{ACCUREV_DIR}storage/”

    ACCUREV_SITE_SLICE = “#{ACCUREV_STORAGE}site_slice/”

    USER = ‘ValidationBot’

    PASS = ‘password’


    def load_trigger_data(file)

    xml = ”

    File.open(“#{ACCUREV_SITE_SLICE}#{file}”) do |f|

    xml = f.read

    end

    Crack::XML.parse(xml)

    end


    def twitter_get_auth(user, pass)

    httpauth = Twitter::HTTPAuth.new(user, pass)

    Twitter::Base.new(httpauth)

    end


    def main()

    hash = load_trigger_data ARGV[1]

    if hash["triggerInput"]["stream1"] == “Validation” then

    changePackageIssue = hash["triggerInput"]["changePackages"]["changePackageID"]

    update_string = “Issue #{changePackageIssue} just promoted to stream #{hash["triggerInput"]["stream1"]}”

    twitter = twitter_get_auth(USER, PASS)

    twitter.update update_string

    end

    end

    main if __FILE__ == $0

    This script is very simple. It makes use of the Ruby gems crack and Twitter. Essentially, the script takes the XML trigger file (provided as ARGV[1]) and loads it into a hash. Then the script checks to see if the stream being promoted to is our Validation stream. If so, it uses the twitter gem to update the status on our ValidationBot account.

    Something's a-Twitter in my backing stream

    A more useful example is to provide a link to the AccuRev Web UI Issue screen.

    Something's a-Twitter in My Backing Stream

    This way, anyone subscribing to the feed can click on the link and open the issue in the AccuRev Web UI.

    Something's A-Twitter in My Backing Stream

    This is a small example of how AccuRev triggers can be used to increase communication on your team. Testers can follow the ValidationBot account and be notified via their favorite Twitter client whenever an issue is promoted to the validation stream and they need to begin work on it. This could obviously be extended to include additional information (actual file changes, for example). That is, of course, if you can fit it into 140 characters. :)

    For more information on AccuRev triggers, please see the Administrator’s Guide.