Notes taken from “Nokia Test: Where did it come from?”[1] (ScrumInc – Jeff Sutherland )
Full Transcript – Jeff Sutherland speaking for 6 minutes (watch video)
Everyone wants to know “How well are we doing and how do we get better?”
The teams want to know that, the leadership wants to know that, the management wants to know that. So, over the years we’ve developed a short test of a few simple questions that can give us a fix on “how well is this team doing?” It started at Nokia in Finland and so has come to be known as the “Nokia Test”.
There are 9 questions now in the Nokia test.
The 1st has to do with iterations, time boxes. Can a team execute a time box in 4 weeks or less? In Scrum we call it a sprint; and are those Sprints stable, the same length, over and over again?[2] If a team can do that, “Yes or No?”, that’s what we want to know for the first question.
The second question has to do with testing, causes an extreme amount of extra work we find that getting software tested at the feature level at the end of the sprints so that a user can use it and give feedback, that is what causes the implementation of software to accelerate, and have the software do exactly what the user wants. We find that delayed testing causes an extreme amount of extra work. Most people don’t realise that. At Palm I asked them to measure it. What is it cost to do testing right now compared to 2 or 3 weeks later? And they found that testing right now, if that takes an hour, 3 weeks later it takes 24 hours. That’s pretty amazing! And they asked, “does that mean it takes 2 years to deliver something we can do in a month?” And the answer is, “yes”. And so the 2nd question is, “do you have your features tested at the end of each sprint?”, “Yes or no?”
The 3rd question has to do with specification. Is there enough information provided by the product owner said that the team understands what they need to do. The pieces are small enough so that the team can easily absorb it, the team can estimate it, they know how to test it, all of these things. “Yes or no?”
The 4th question has to do with the product owner. In Scrum the product owner is responsible for creating the backlog for the team to implement and ordering that backlog; and having one backlog that the team is responsible for executing. It’s amazing how many companies have trouble finding a product owner or they get a product owner and they’re not a good product owner, they are not able to really deliver on the backlog; so “do they have a product owner that is able to deliver good backlog or not?”, “Yes or no?”
The 5th question is the backlog itself. It’s really important that the team has seen the backlog and the team that is going to work on it has estimated the size of the backlog before they go into sprint planning. If they don’t, they get into the planning meeting right at the beginning of the sprint and there’s lots of questions, some of them the product owner can’t answer, so it drags out the Sprint planning meeting and it gets them off to a bad start. So, do you have product backlog that is in a ready state, it’s been looked at, it’s been estimated, all the questions are answered before sprint planning starts, and then, in Sprint planning you can look at it and figure out how much you can take. “Yes or no?” Do you have that, or not?
The next question is the estimation process itself, it’s usually done before sprint planning, in terms of relative sizing, and if a team uses what we call “Planning Poker”, it’s based on “Wideband Delphi Formal Technique”[3] (RAND Corportation, 1950-1960s ) invented for the Department of Defence, if they use that, we find that teams can often estimate 40 times as fast as traditional estimation procedures. It can cut the whole project planning cost by 80% by doing this right, so it’s extremely important. So, “is the team using planning poker to estimate the backlog?”, “Yes or no?”
The 7th question has to do with the burn-down chart. Does the team have a burn-down chart that shows how much work they have left until the end of the sprint? And are they updating that on a regular basis? “Yes or no?”
The 8th question has to do with disruption. We want the teams to take backlog and have that backlog stable during a sprint. Is there anybody coming into the team, changing their priorities, pulling people from the team, or generally causing mayhem? It’s surprising how many teams have this problem, so either you don’t have disruption, or you do. “Yes or no?”
The final question is the team itself. It is the team able to operate in a self-organised state? Does the team pull their backlog? Do they decide themselves how they going to work the backlog? Can they operate without someone assigning them tasks and telling them what to do all the time? “Yes or no?”
When we start with new teams, even though they’ve been doing
Scrum for a while, and we ask them these questions, we find that they usually
score about 40%, and as we work to improve the answers to these questions and
the score goes up to over 60%, they usually double their output and if it goes
up over 80% they often triple it; in fact we’ve had many companies within
OpenView Venture Partners[4] (OpenView Venture Partners, 2007) that have tripled
their output in the first 2 or 3 sprints just by looking at this test so it’s
very, very useful
[1] (ScrumInc – Jeff Sutherland )
[2] “The purpose of the fixed interactions is to establish a “heartbeat” of the team which leads to higher performance. It is also to give the Product Owner a predicable roadmap based on team velocity.” – Jeff Sutherland
[3] (RAND Corportation, 1950-1960s )
[4] (OpenView Venture Partners, 2007)
