At the XP.BE user group meeting hosted by ERG Transit Systems, we ran a workshop on “Management metrics”.
The people at ERG wanted to introduce some metrics to get feedback on their process improvement efforts. However, metrics can be dangerous. “You get what you measure”, but what you measure may not be what you intended.
We used a workshop format designed by Jason Gorman and Duncan Pierce to design, break and improve metrics. The session went like this:
- we formed 4 teams
- 3 people from ERG presented 4 requirements for metrics, one for each team. For the rest of the session, they acted as “onsite customer”
- each team designed and presented a metric to satisfy the requirement
- each team received another team’s metric and tried to “game it”, to subvert it to reach the opposite of the desired effect, or to score as high as possible with the least amount of effort. This was the fun part.
- each team then improved their metric, taking into account the possible “misuse”
- our customers performed the acceptance test. Each metric had to pass two tests:
- Would our customers be able and willing to implement this metric tomorrow?
- Would the developers want to work in a team that used this metric?
3 out of the 4 metrics passed the user acceptance tests. The fourth might too, with a bit of work. Not bad for 90 minutes of work!
Some things I learned about metrics:
- No metric is fraud or misuse-proof, therefore people have to want to use the metric correctly. If people want to game the system, they can be very creative.
- Metrics that take a noticeable amount of time and work to collect will not last long
- It’s better to aggregate data, because errors in the individual items tend to cancel each other out. E.g. it’s easier to track the estimates of a whole release, rather than the estimates of each individual feature
- It’s better to look at team results, rather than individuals, because of the aggregating and because that motivates people to work as a team. This is related to the “Reward one level up” rule I first heard from Mary Poppendieck.
I ‘d like to align metrics with the “Throughput Accounting” measures. I’ve added an “Idea for Session” about this topic on the Agile Open site. How about you?
More useful info and books on the wiki.
Jason and Duncan will host this session at SPA2006. Highly recommended!
Update 20/02/2006 Nico Mommaerts has blogged about this event too.
Tags: SPA 2006, Agile Open 2006, metrics, XP
