mcLEAN Practically Agile :: Agile Metrics Report

Some great figures in this document: agile projects are 37% faster to market, experience a 16% increase in productivity and with no discernable impact on quality.

The publication has an authoritative source so it carries a lot of weight. I've just got a bit of a dilemma though, and Melinda if you read this I'd like to solicit a response from you if you please.

The study makes the distinction between agile companies that use Rally software and those who don't and points to significant additional advantage for the former. Unfortunately it fails to pin point exactly how Rally makes this difference - which of its functions realise the benefit - and surely there must be some dynamic related to how its been rolled out and how its used on a day to day basis. A great marketing message for Rally - but I have an observation:

The agile manifesto promotes "Individuals and interactions over processes and tools" but here is a study that seems to suggest that the introduction of a tool enhances agile in a general sense. I guess there might be room for a tool making a little difference - but considering non Rally projects being 37% faster to market and Rally projects 50% faster to market i.e. about 33% faster again seems to run contrary to the manifesto. Do these figures challenge the manifesto? Is it a problem if they do? Is the manifesto wrong? Or is Rally pushing the marketing envelope a little too far?

Also, we don't know from the report if the other agile teams used any tools other than Rally - tools that are worse perhaps?

Also, we don't know from the report if the other agile teams were just worse at doing agile - what control is there to eliminate that and give us the confidence that it was definitely the use of the tool?

I'm happy to see some agile metrics and I don't have anything against Rally - but I'm concerned that its a bit too easy to challenge this paper.

Posted to:

Main Page

News

Comments

Post a comment

Re: Agile Metrics Report

by MichaelMah on Wed 26 Nov 2008 15:18 GMT | Profile | Permanent Link

I'm Michael Mah and I was the author of the findings contained in the report cited here. First I'd like to thank the author for the comments in this post. They are well thought out and well articulated, and I'm happy to offer a response.

First, this is a second-generation analysis of agile projects from the group of companies that are represented. Initially the study looked at 20+ projects from 5 companies, and I've presented the results at several conferences including Agile 2008, Better Software, Agile Development Practices, and in webinars via the IT Metrics and Productivity Institute and the Cutter Consortium. There were definitive patterns on faster time-to-market, productivity, and defects/quality compared against our QSM Industry Database of 7,500 projects. Two of the five agile companies agreed to be named: BMC Software and Follett Software. The former took a SCRUM approach and the latter an XP approach. Their results were significantly better than the other three of the group. BMC used Rally tools, while Follett did not.

The second incarnation of this study added 4 more organizations - all using Rally - who happened to sponsor the study. It just so happened that the schedule, productivity, and quality numbers were also excellent. I'd say that having a technology framework for maximizing communication with common knowledge resident on the platform was a key attribute, but I would not simplify this per-se by saying it was the only factor behind the numbers. There are several at play, but it just so happens that the Rally group shows strong metrics, and that the framework helps teams become as effective as they can be.

I hope this helps. I'm happy to offer additional insights.

Re: Re: Agile Metrics Report

by mcLEAN on Thu 27 Nov 2008 13:38 GMT | Profile | Permanent Link

Michael - thanks - and apologies if I misrepresented your authorship.

I'm comfortable with the assertion that a tool that facilitates communications within the extended team will help – I’m still a little surprised that it should have such a large impact – the “several” other factors are what I am interested in as they must surely contribute significantly too.

Presumably similar benefit should be possible for teams who employ similar comms tools other than Rally – some control data relating to that would interest me too.

I would also assume that with the introduction of any such tool the benefit reaches an optimal level before its integration with the team becomes an overhead – we’ve all seen this the world over with email. So all the teams included in the study must be using Rally optimally or near as in order to attain such large benefits.

I’m hoping there will be more studies soon to continue stimulating this subject and opening these questions to scrutiny – its all very healthy for agile.

Re: Re: Re: Agile Metrics Report

by MichaelMah on Thu 27 Nov 2008 14:30 GMT | Profile | Permanent Link

You are most welcome. One additional comment about "Group 2." There was a definite emphasis on time-to-market; seems we are a schedule-driven culture for sure. And while the defect stats were "just ok," there were not nearly as impressive as Follett and BMC. This group took the schedule gains and less of the quality, it seems to me.

I also believe there is more to learn, and QSM Associates is positioned to continue this work as more data comes forth through our consulting and research, and for those agile companies that measure - many of whom are using our QSM SLIM framework. It is indeed all healthy for agile.

By the way, I published a much more comprehensive analysis on agile through a report with the Cutter Consortium (I'm the Benchmarking Practice Director). It's entitled "How Agile Measures Up, and What It Means to You." It goes into far more specifics (nothing really on tools) and includes interview excerpts with Kim Wheeler of Follett and Mike Lunt of BMC. Mike is also a contributing autho; his section is entitled "The Secret Sauce."

It's available at www.cutter.com. Keep the conversation going! Please visit my blog for more - www.optimalfriction.com

Trackbacks

TrackBack URL:
http://blog.viocity.com/blog/_trackback/3994284

No trackbacks found.

Post comment:

	Receive comment notifications for this article
Subject:
Comment:

Comment verification:

Please enter the text you see inside the graphic to post your comment:

You are not currently logged in. If you would like your user information to be displayed with your comment, please enter your login information below.

Username:
Password:

If you would like to post contact information on your comment, please enter your information into the optional fields below:

Contact information:

Name:
URL:		example: http://yourdomain.com
Email:

Please note: email will not be displayed on the site, only for the blog owner. If logged in, URL will only be used.

FREE Online Agile Tool

CardConversationConfirmation.com is an online tool for managing User Stories, the detailed conversation that follows and the satisfaction criteria that defines how they should be delivered.

It's free to use and you can start now by registering here.

Agile Books

If you see a book here that interests you and you decide you want to buy it - if you use this link it will go some way to paying the hosting bill for this blog - thanks.

see a full range of books recommended by mcLEAN Practically Agile

See my new Book Review section

Amazon.co.uk Widgets

Software Development Lifecycle

Part 1 - Introduction

Part 2 - Card Conversation Confirmation

Part 3 - Agile Workflow

Part 4 - Key Performance Indicators

Part 5 - Governance - Completion of Artefacts Introduces Agile Benefit