Tweet: From Scrum to Kanban: 50% shorter lead time, 10% fewer bugs (even fewer critical), and improved productivity
Tags: Software process, Scrum, Kanban, Document analysis, Interviews
Paper: Dag I.K. Sjøberg, Anders Johnsen, Jørgen Solberg, "Quantifying the Effect of Using Kanban versus Scrum: A Case Study," IEEE Software, vol. 29, no. 5, pp. 47-53, Sept.-Oct. 2012, doi:10.1109/MS.2012.110
Background
B1. Many companies switched to Agile/Scrum but little is known about the effects
Problem: Lack of empirical data on Kanban and other agile methodologies
Method
M1. Studied process quality from work items and data collected in the project tool used at company (Microsoft's Team Foundation Server)
M2. Studied how process (Scrum or Kanban) and type of work item (bug or project backlog items) affected lead time, productivity, throughput and quality.
M3. Code churn used as indicator of effort since no reliable effort data available
M4. Bugs were classified and weighted according to severity: blocking (weight: 8), critical (4), moderate (2), minimal (1)
M5. Quantitative analysis was complemented by 4 interviews with key people.
Results
R1. Average lead time (per work item) decreased 50% when using Kanban. Standard deviation in lead time was also much smaller with Kanban.
R2. Average number of (weighted) bugs per quarter decreased 10% in Kanban projects. For the most critical bugs the reduction was larger, 26%. However, this can also be because more and better testers were employed.
R3. Productivity decreased for fixing bugs but increased for non-bug work items in Kanban projects. An overall increase.
R4. Productivity decreased for fixing bugs but increased for non-bug work items in Kanban projects. An overall increase.
R5. All interviewed employees preferred Kanban to Scrum.
R6. Timeboxes in Scrum were considered artificial, lead to underestimated work items and made it hard to use resources efficiently within sprints.
R7. Sprint startup meetings were perceived as waste.
Limitations/Threats
L1. Better results with Kanban can be because it was used after they already had learned one Agile process (Scrum).
L2. Different implementations of Kanban and Scrum might make the difference so comparison/generalization between contexts might be hard.
Raw numbers
D1. Company has 330 employees of which 100 are developers in 10 teams
D2. Company used waterfall process 2001-2006, Scrum 2007-2010 and then switched to Kanban.
D3. Data in study only taken from 2009 onwards after agile transition phase considered "done"
D4. Average lead time was 5/7 days (Bugs/Non-bugs) with Kanban and 12/14 days with Scrum.
D5. Standard deviation in lead time much smaller for Kanban (5/9 days) vs Scrum (17/20 days).
D6. Bugs were to 70% detected internally and 30% detected externally (by customers).
D7. Standard deviation in weighted number of bugs per quarter decreased from 47% of the average (Scrum) to 30% of the average (Kanban).
D8. Productivity (when measured in work items per person) decreased 21% for fixing bugs but increased by 73% for non-bug work items.
D9. Productivity (when adjusted by churn) decreased 11% for fixing bugs but increased by 21% for non-bug work items.