Sep
19

Don’t outsource your mouth, ears and brain

Helpdesk by Scott Feldstein

A call to the support desk

I recently bought an extra service from a service provider who shall remain nameless. I logged into their online account management system, selected the options I wanted, clicked CONFIRM and was promised same-day activation of the options. I got an email confirming that everything I requested had been done. It should work now, but it didn’t.

By the end of the day, the features still didn’t work. The next day, still nothing. Checked the online system: the options were listed as ‘Enabled’, but still they didn’t work.

I called the company’s toll-free support desk. After navigating through three levels of menus of the automated call system, I got through to a human being. I explained the problem. The support technician asked a few questions and had a look in the affected system. The problem was clear: “Despite the fact that their system says the feature is enabled, they haven’t actually enabled it. I’ll send a request so that they enable the option. That usually only takes a few minutes, maximum one hour.

A short while later everything worked as it should. And everybody lived happily ever after. Except…

There’s something wrong here…

The call to the support desk had been fast and efficient. The problem was solved. Why did I feel I had missed something?

I took a while to find the cause of my uneasy feeling: the support technician used the word “they”.

I thought I was calling the service provider. It turns out I was calling an independent support desk who weren’t a part of the company (or who felt that they weren’t part of the company). For them, the work was done the moment the call ended. If the support desk has decent incentives in place, this technician got rewarded for efficiently solving a customer problem.

As a customer (and if I were a manager at the company) I would like to see this call as the start of the work.

Why did I make this call? Because the online account management system says it enables features, but doesn’t actually enable them. Why? I don’t know, but I’d like to know. Does it do this consistently? If yes, that’s going to lead to a lot of support calls. The statement “…that usually only takes a few minutes…” indicates that I wasn’t the first customer to call with this type of problem.

Actually, the online account management system does do something right: if you enable the feature, you will get billed for it. Billing for a service you don’t deliver makes customers unhappy. How often does this happen? How many complaints do we get about that? How many customers have we lost because of that?

I doubt any of those questions get asked. The phone’s ringing again. Here comes another complaining customer. No time to waste, we have to hit the the helpdesk KPI targets!

Wasted brainpower

Companies spend a lot of money on salespeople, because they are the “face” of the company and they bring in the money. Support personnel is as much the face of the company and they have to represent the company when the going gets tough: when the customer has a problem or is unhappy. That’s when you get to see the “real” company. That’s when the company gets great information about unclear features, about what customers really need and about problems with their processes.

Support people are the mouth and ears of your company. They should be trained and rewarded accordingly. They can be the brains of your company, but only if you give them the time and means to find the root causes of problems. And then only if you have the courage  to tackle those root causes.  But helpdesks are seen as a cost center: “we want to get rid of that complaining customer as fast as possible, for the lowest possible cost.”

Well, providing poor support is a good way to get rid of customers. That certainly lowers the cost of the helpdesk. But then you have to pay more for salespeople, to attract new customers.

You wouldn’t outsource your ears, mouth or brains, would you? Why would you outsource support?

What can you do today?

  • Make sure you thank everybody who brings bad news. Thank you for calling in with your complaint! Thank you for finding this bug in my code!
  • Perform root cause analysis on each and every bug report, customer complaint or support incident. Implement Poka-Yoke in your development team or implement ITIL Problem Management in your operations team. I know you think you don’t have enough resources to do this for every case, I’m a bit extreme in these things. The goal is to eliminate the need for a helpdesk and to make all our customers into salespeople by providing an insanely great service or product.
  • Involve support people in root cause analysis and finding solutions, not workarounds! They know the product inside out, they know the customer and are born/trained problem solvers.
  • Let support personnel create high priority, high value requirements for the product because they know what it takes to retain, satisfy and delight customers.
  • Involve the support team in defining the product/service from the start to create supportable products.
  • Provide time, training, tools and techniques to perform root cause analysis for everybody in the company.
  • Reward appropriately. Reward for customer satisfaction, customer retention, problems avoided, quality increased.

What have I learned today? Be wary of that company and their online tools. They seem more interested in billing me than in providing a service. And I doubt it will get better, because they don’t seem to learn.


Photo by scottfeldstein

Aug
22

Back to the floor and back again to the boardroom, no wiser

Genchi Genbutsu on TV

Many years ago there was a very interesting and infuriating series on BBC tv called  “Back to the Floor“. The premise was simple: let a company director work different jobs “on the floor”, follow them with a TV camera and see what they’ve learnt. I didn’t know the term “Genchi Genbutsu” yet, but I thought it was an excellent idea.

And it made for interesting television:

  • most executives weren’t very adept at performing tasks, to the amusement of their employees who had to teach them
  • the executives received a veritable barrage of problems that people experienced on the floor. The executives were invariably suprised, these problems never reached the boardroom. They had no idea.
  • the executives returned humbled to their boardrooms to tell their fellow executives of their adventures and all the obstacles they had to overcome just to get some work done

Each episode always ended with the executive ordering that actions be taken to solve the problems they had encountered. And so, Betsy and her team finally got someplace to brew a cup of tea. And everybody was happy.

And what have we learned today?

Infuriatingly, almost all of the executives only experienced “Single loop learning“: they had seen some problems and taken action to solve those problems. And then they went back to the order of the day. Nothing had really changed, except for Betsy.

Very few executives asked the uncomfortable questions: “why is it that we never hear of those problems?“, “why is it that these problems don’t get solved?“, “why do people have to overcome so many obstacles to get their work done?“.  When one executive asked questions like this, you could see the other managers squirm and try to avoid being blamed. Only a handful of them took action so that the whole management team went back to the floor regularly.

No manager that I can remember went so far as to look for systemic causes of the problems. That’s probably a bit too much to ask in only a week, and a week that’s full of “real work”. There’s no time to think, we have to overcome all those obstacles!

Do you go back to the floor? What do you see? What do you do about what you see? And what do you do about the causes of what you see?

What have you learned today?


Meeting room stencil graffiti by Richard Rutter

Jul
21

The Paper Process

I’m sorry, I can’t help you because the system is down

How many times have you heard this apology? Often, the apology is “The system is down again.” In some places the system is always down around certain times and you get the same apology every day.

Why are we suprised when systems are unavailable? There will be bugs, mistakes and scheduled downtime. We should not be suprised the system is down, we should expect it.

Some time ago, an IT architect asked me during a job interview if it was possible to build reliable systems out of unreliable parts. My response:

The only way to build a reliable system is to build it out of unreliable parts (or systems)

If you want to build a reliable system, you have to be aware that all its subsystems are unreliable. This allows you to take appropriate measures, like building in redundancy.

We need a paper process

A smart manager I once worked with insisted that every business-critical automated process also required a backup “Paper Process”.  The Paper Process defined how the work would be done when (not if) IT systems were down, with nothing more complicated than pen and paper (and a battery-powered calculator if really needed). When systems were unavailable everybody knew what they had to do to keep the business ticking over as if nothing happened. They also knew how to catch up when the systems were available again. In this case, it was clear who the bottlenecks were, so the other people subordinated by entering the backlog of data on paper into the system. Did I mention that this department used less automation than other similar departments, yet had a better track record of delivering on time,  was more efficient and brought in more money?

Defining an alternative Paper Process was relatively easy, because we really understood the real business requirements of our customer. Since then I use this as a test of my understanding of the real requirements: could I implement the requirements with nothing more than paper and pencil? If you have real requirements (and not a solution in disguise) it’s easy to define several different implementations.

Since that project I learned more about processes. If I had to do this project again I would do it the other way round: implement the Paper Process first and ask what, if anything, we would need if the Paper Process broke down. Maybe a whiteboard is enough.

Jul
18

Avoid Change

The thing I hate most about myself is that I don’t want to change

The first XP book was very important for me. It changed the way I work. It changed my life.

I still apply the practices to improve the way I work. The principles have helped me to make my work more fun, sustainable and successful. The values have helped me to do worthwhile work that delivers value.

But “Embrace Change“? I don’t like change.

How to avoid change

I avoid change on my projects by

  • Really understanding the needs of customers, stakeholders and users. Those needs change from time to time, but not very frequently if we really get to the core of what these people value. I get accused of doing Big Upfront Design or Analysis, even in companies seemingly doing Waterfall. Why waste so much time upfront, can’t we start building yet? Customers don’t know what they want anyway, so why try to discover what they need?
  • Not committing to decisions too soon, applying Real Options. If we commit later by delaying decisions to the last responsible moment and gather more information, we need to come back on fewer of our decisions. I do get called a procrastinator. Real leaders take charge, take decisions!
  • By not being distracted by useless technology churn. Simple, reliable and known tools and technology in our toolkit let us concentrate on adding value. I do get called a dinosaur, not up to date with the latest framework or fad du jour. Real geeks only work with the latest 0.1 version of technologies that are so complex that we can spend months unraveling their intricacies (and bugs). I need a well-padded resume with all the latest technologies to get that next job when the shit hits the fan on this job.
  • By making sure we consider supporting processes and users in our value analysis. All too often we focus on the shiny business value generating processes but forget that these can’t work unless people can perform the unglamorous supporting jobs that make these value streams possible. I do get called a Waterfall analyst, who wants to explore every nook and cranny of the system before we can start building. Real Agile teams deliver Business Value, cash quickly. Why waste time with non-business value-adding processes like getting the product into the hands of the user or managing the system to keep it relevant?
  • By making a plan (not a schedule) of the goals we want to achieve in the future so that we have a guiding vision when we travel the path. I do get called an old-school project manager who wants to control and stifle the creativity of his “resources”. You can’t be agile if you have a plan.

Projects with fewer changes are a lot easier and less stressful than those with many changes. These projects implement Heijunka. They deliver value surely and steadily. I like that.

Embrace risk reduction and new information

I welcome any change which reduces risk. We continuously identify our main risks and find ways to avoid them. For example, we discover that there’s simple alternative way of doing something. We can use that as our backup strategy if the planned way of doing the work doesn’t get ready in time. For example, we first implement a very basic implementation of a business process and iteratively refine it, so that we’ll always be ready to release by the deadline. Of course, we have to use that extra time that Real Options gives us to actively seek out information and reduce risk.

Embrace increased flow

I welcome any change which can increase flow, get us to release faster and get the cash flowing sooner. For example, we can start by statisfying a subset of stakeholders and users. We can focus on on value stream at a time. We can mercilessly pare down what is really needed to achieve goals. We can get rid of wasteful checks, approvals and delays. We increase quality to decrease changes due to rework.

Embrace value increase

I welcome a change that increases value. For example, “Exchange Requests” let the customer increase the value of our work. If the customer wants to add a feature to a release, they first have to remove an equivalent amount of work from the release. The customer will only perform the swap if the new feature has more value than the one swapped out. So, for equal (or lower) cost, we deliver a product with higher than initially expected value.

Embrace cost and investment reduction

I welcome a change that reduces cost or investment. I welcome a new, simpler way of achieving a goal. I welcome reducing scope as we discover that we can simplify business processes. I welcome a reduction in “dimension” as we discover that we can satisfy user needs (for now) with less exhaustive or refined implementations. I welcome a way to reduce the amount of work in process.

I’ll get me coat then…

Can I be Agile if I don’t Embrace Change? Can I be Agile if I value people and their interactions AND processes? Can I be Agile if I insist on customer collaboration AND a contract?  Can I be Agile if I only want to respond to beneficial change AND follow a plan? Can I be Agile if working software is not the solution or only a small part of the solution? Can I have my cake AND eat it too?

Where do I go to hand in my Agile badge?

If anybody needs a Waterfall coach, let me know. I can change. I’m agile 😉

Jul
16

Software Development Takt

Takt time

Takt time is the rate at which customers buy a product. Takt time determines the speed at which a Lean manufacturing runs. For example, say we sell 576 units of product A. If we have a 8 hour shift, we need to make 72 units per hour, or 1.2 units per minute. We have 50 seconds per unit.

Every step in the production process must now deliver their contribution to the product in 50 seconds.

What if there’s a step that can’t deliver in 50 seconds? Then we have a bottleneck. We’ll produce and sell less than we could. We apply the 5 focusing steps to increase the bottleneck’s capacity so that they can deliver in 50 seconds or less.

What if there’s a step that can deliver in less than 50 seconds? They still deliver every 50 seconds, even if they have to slow down. If they delivered more frequently, we’d build up lots of work in process as other steps can’t use its output faster than once every 50 seconds. Or we might build unsold goods.

Slowing down? Isn’t that inefficient? Yes, but we’re not optimising the efficiency of a single step. We’re optimising the whole value stream. If a step has a lot of spare capacity, this could be used to work on another product, but only if it doesn’t endanger the delivery of the first product.

Using Visual Management, the value stream tracks actual production rate regularly. The steps won’t run perfectly as clockwork, there will always be some variation. The production rate displays allow everybody to react (speed up, slow down a bit) to keep to the takt time. The takt time works as a metronome that synchronizes musicians.

Takt time isn’t (always) the drum beat

In Theory of Constraints, the Drum-Buffer-Rope system synchronizes the speed at which each step in production runs. The “Drum Beat” is the rhythm, the steady pace at which the slowest step (the bottleneck) runs. The rest of the system works to that beat. Again, no use producing faster than the bottleneck because we’ll only create more work in process.

The two concepts are quite similar and I sometimes confuse them. There’s one clear difference:

  • takt time is customer-focused, outward looking
  • drum beat is bottleneck focused, inward looking

If the customer is the constraint, a base (if not outspoken) premise of Lean, takt time equals the drum beat and drum-buffer-rope implements customer pull.

Takt time in software development

What is the equivalent of takt time in software development? If you’re Microsoft, you know approximately how many copies of Microsoft Office are bought every day. You could run your disk duplication and packaging plant at this takt time. But that’s not what I want to talk about.

Takt time = the rate at which you release a new product (version) to users

How often do you put new, valuable features in the hands of your users? Every day? Every week? Every month? Every quarter? Every year?

The logic of slow takt times or how to kill all your Agile teams with one simple decision

Most teams I work with, including Agile teams, have long release cycles. Long by my standards. Some of this is under control of the development team, but the release cycle is often dictated by other concerns. Often, operational teams (systems engineers, DBAs, support) reduce the number of releases.

Operational teams are often stressed, in fire-fighting mode and jerked around as projects miss deadlines and make last-minute changes. To reduce the load and ease the pressure, they reduce the number of releases. Fewer releases means more complex and error-prone releases, as more features are batched up in one release. These releases require more effort and have more issues. Fixing and patching those issues requires more work. Business people, under pressure to generate more value faster push harder to get features out of the door. Corners get cut, features get released under the guise of fixes. More work. The logical reaction is to reduce the number of releases again. Which makes the problem even worse. And so on.

Reducing the number of releases, increasing the overhead of releasing is the perfect way to demoralise and ultimately kill any Agile team. Why should we define short iterations and releases, if it takes so long to get something in production? Why should we collaborate tightly with customers if the software is outdated by the time it gets into the hands of users? Why should we focus on business value if potentially valuable features gather dust in an endless release process? We might as well switch to waterfall.

It doesn’t have to be this way. The first step is to regain the trust of the operational teams: release on time, deliver great quality code, involve operational teams in planning and retrospectives, treat members of operational teams (who are often involved in many projects) as members of your team: listen to them and optimise the whole value stream.

Once we’ve established trust and created one team, we can start to talk about improving and going faster.