Abstract
For over a decade now, 'agile' has been the in-vogue development term. Surrounding the associated methodologies there has been something of a, 'not your father’s development method' attitude. But as methods have matured from their original youthful exuberance, a range of practical processes have become established. So, what are the differences between each of these process variations, and which would be better for your organization? In this paper, we will simplify the issue and consider just two popular methods: Scrum and Kanban.
Waterfall vs. Agile
Before we discuss these two Agile methods, we should take a step back and consider the shift from traditional methodologies to more modern and Agile ones. They have been seen as something of a counter-culture to the traditional Waterfall development method — which has, all too often, been associated with failed projects. Many of the proponents of Agile development would have you believe that the Waterfall framework itself was to blame, but it would be unfair to place the responsibility squarely at the feet of the process. All methods have their pros and cons, with varying applicability to circumstances and environments. So let’s start with how do Agile methods compare to the Waterfall model?
Length of Development Iterations
One of the gospels of the Agile bible (were there such a thing) would describe the importance of simplicity. Traditional Waterfall methods have evolved to manage relatively large and lengthy projects and to do so, more rules and procedures are required to keep control of it all. The Unified Process grew out of the dissatisfaction with traditional large-project management methods, but still prescribes far more than recent Agile methods in an attempt to accommodate larger projects. In this way, the process actually adds complexity to the project, while trying to control it. In contrast, Agile methods try to get out of the way and let the work get done with less overhead, not more.
Task Scale
Given that larger tasks are harder to manage (and therefore, are more error-prone), Agile methods take larger tasks and break them down into smaller, more manageable, and more risk-averse units. Smaller pieces require shorter development iterations and shorter development iterations are easier to manage. Additionally, the human mind (both individually and collectively) is limited in its ability to constantly conceive of, and contemplate, large complex problems, which Waterfall would have us do. Agile, on the other hand, breaks the problem down into smaller pieces, each more easily understood and processed than the whole.
Development Cycle Scale
So, Waterfall methods have one (or a few) large development cycle(s) in which managers try to keep control of a large plan and see that it stays on target. Agile, by contrast, requires smaller bites of the cherry, where the focus is on managing each mouthful, not the entire meal. This is true for both Scrum and Kanban, as we will see.
It is important to realize that with Agile methods, it is not just the software that is written in smaller pieces; the majority of the development cycle is decomposed into smaller elements, from detailed specification through design, coding, and testing. Critically, testing involves regression testing (often automated) to ensure that each new addition does not break what was done before.
Illustration 1: Rudimentary waterfall model showing single large cycle
Illustration 2: Rudimentary Agile model showing similar activities in small iterations
Requirements and Change
Waterfall methods tend to result in a culture where large sets of requirements are bundled together and tossed over the wall to development. And so on down the line so, consequently, the level of involvement of each person varies with the phase of development.
With Agile, requirements are prioritized into a 'stream' of needs from which work is assigned piecemeal, with all team members from stakeholders to testers being involved with each new iteration. This constant involvement fosters stronger relationships between customers, those in product management roles, and the development organization. It also becomes easier to incorporate change, as the stream is more adaptable than the whole requirement set delivered up front. By including all players at all times, change becomes something endorsed by all, not determined by a few.
To return to the analogy — Waterfall requires force-feeding of the whole plate at once, Agile allows one spoonful to be taken at a time (and voluntarily).
Status Monitoring & Process Improvement
With Agile, progress is easier to monitor because the successful completion of each small element is easily seen, whereas with the Waterfall method it's harder to tell what works fully and what doesn't — especially as acceptance testing does not usually come until all coding is complete — and so visibility into the status of the project is much harder to achieve. While it is easier to predict, or in some methods ensure, the end time for smaller elements in Agile processes and the overall completion time for a large collection of smaller pieces in Agile may be equally difficult to estimate. This may require a change in culture, to one where managers no longer expect or desire bulky deliverables and their associated estimates. Instead, they appreciate the incremental delivery of tested, working solutions from which they can see early on whether components work, and not nervously wait until the end for the big reveal.
With frequent visibility of status comes the ability to review successes and failures and use that knowledge to improve the process in real time. Agile methods prescribe process improvement during the project itself, which is possible due to the smaller packaged and independent nature of tasks compared to Waterfall, in which process reviews are not easy while the project is still in progress. As we will see with Scrum and Kanban, not all process improvement methods are the same within the Agile family.
Kanban vs. Scrum
On that note, how do Scrum and Kanban actually differ? My father always says, “Doing something that way will take 7 days, whereas doing it this way will take a week,” and there is some truth in that where Agile methods are concerned — the similarities are more important than the differences.
It should be said that many concepts of one Agile method can be, and often are, borrowed by another, and thus, projects following a single pure Agile method are not common. It is probably wise to take what you need from each method and make it work. Constant process review and improvement (see below) will help identify the ideas that are working and those that are not. Let's look at some of those concepts and how they vary between these two.
The key differences between the Scrum and Kanban methods are:
- Roles: Defined versus open
- Task scheduling: Rigid sprints vs continuous ongoing work
- Task sizing: Smaller tasks vs tasks of any size
- Control of WIP: Sprint vs work state limits
- Change friendliness
- Meetings and visible progress displays
- Suitability for maintenance work
Roles & Teams
In general, Kanban offers a flexible approach that makes it easier to adopt by teams who are more familiar with traditional Waterfall methods; one of these is greater freedom in the assignment of roles. While Scrum requires specific functions to be performed by individuals assigned specific roles, Kanban does not. Scrum defines three roles: product owner, scrum master, and team member. Of course, other roles can be used in addition to these, but without these three, you are not doing Scrum. With Kanban, each user may continue with their existing role until the change in activity or responsibility presents itself organically and roles evolve.
We will see this kind of flexibility on the part of Kanban repeated as we examine other differences, which can be both an advantage and disadvantage. On one hand, it makes it easier to adopt by teams that have not been agile before. But on the other hand, there are more process decisions to make. If the current process is not working well, Kanban offers fewer 'better' ideas to replace those that are failing.
How this Plays into Processes
During each sprint, (the Scrum term for iteration) in a Scrum process, individuals work as a team to achieve common goals. This is necessary because the sprint is tightly time-bound, requiring all work, regardless of its nature, to be complete by the end of the sprint. Roles are less rigidly defined and whatever needs to be done, must be done, by someone! Clearly, there are limits to this skill generalization, but regardless of formal title, team members on a Scrum project are encouraged to help others and so team members with more rounded skill sets become more valuable. Also, team cohesion is a must, and experienced software engineers often do better in a Scrum environment than junior programmers do.
Kanban projects allow (but do not force) individual engineers to work on their own activities with greater independence from their colleagues, especially in regards to the deadlines of others. This has much in common with traditional development methods, so is easier to adopt initially.
Task Scheduling/Sizing
Scrum requires that a small set of tasks be started and completed together, within a small time window (known as a sprint). This does not come naturally to many engineers and so it is, again, less straightforward to initiate than Kanban where tasks can vary in duration and therefore their start and end times do not have to be synchronized.
With sprints usually lasting between 1 and 4 weeks, tasks must also be limited in their size. Otherwise, they just could not be completed within the sprint — and all tasks in the sprint must be completed for the sprint to complete. Consequently, work is divided into rather small tasks, and estimating how long any task will take becomes critical to decomposing the tasks successfully. Because Kanban does not prescribe fixed sprints, tasks can take any length of time and therefore be any size (which means that estimating the size of a task is less critical).
Scrum, therefore, controls the amount of work being actively done by constraining a sprint to a defined amount of effort. More accurately, this is the number of story points, an arbitrary measurement of how 'big' a piece of work is relative to all other pieces of work. A sprint can fit X number of story points each time. But Kanban, doesn't have a sprint, so how would a Kanban project avoid throwing everything into the process at the same time and becoming Waterfall? The answer is Work-In-Progress, or WIP.
Work In Progress (WIP)
Regardless of the Agile process, it is a good idea to control how much work is being undertaken at any one time. Remember, Agile is about biting off small amounts each time. Scrum has an in-built mechanism for limiting the amount of work in progress at any one time: The sprint. Scrum requires that the amount of work selected for a sprint and the length of the sprint be precisely defined. The amount chosen is based on the team's 'velocity,’ or the amount of work they are typically achieving in each sprint.
Velocity
This seems like rather a circular definition and to some extent it is, but that's OK — process improvement is all about feedback. The basic rule is to only assign the work that can be done in a single sprint. This seems ridiculously obvious, but the trick is to know the amount a team can actually do, which can only be determined through historical trends. How well has this team done so far? New team? No worries, use estimates from previous similar teams on similar projects. If the sprint fails to complete on time, adjust the velocity and thus reduce the work assigned for the next sprint, and so on. This is an example of the constant process improvement that Agile projects should be pursuing.
Although tasks within a Scrum sprint can be totally unrelated, in practice most teams choose tasks with a common theme. This not only gives the sprint as a whole some degree of purposefulness, it can cut down on the areas of the existing code that are touched and therefore reduce both risk and necessary regression testing.
Kanban Work Items
So, in a Scrum project, the amount of work chosen for a sprint and the sprint duration are fixed and immovable (although through process improvement, these may change from sprint to sprint.) Kanban, by contrast, has no defined start or end points for work items. Each item can start and end independently from any other item, and work items have no pre-determined duration for that matter. Instead, duration estimates are made for each task, and work is started when effort becomes available. Control in Kanban is imposed by controlling how much work can be done in each work state. States may be defined in any way, but examples might be as simple as 'Pending,’ 'In Progress,’ and 'Complete.' In reality are better with a greater granularity of functions such as, 'Analysis,’ 'Integration,’ 'Test,’ etc. So, in Kanban, the control gates exist between each state with nothing moving forward until something clears ahead.
An analogy using traffic would have a small number of cars entering and exiting the interstate together (Scrum) or each car entering the interstate separately and moving a few miles only when the road ahead becomes clear (Kanban). By contrast, with Waterfall, all the traffic for the day enters the interstate at the same time and each car competes to move until all the cars have finally exited the interstate.
Illustration 3: Scrum: Work items go through a sprint together
Note: Each sprint completes before the next starts.
Illustration 4: Kanban: Work items make their own way and wait for each next state.
Note: All states are active at all times
WIP Inefficiencies & Limitations
One consequence of a sprint is that there will be regular builds which, in theory, could be released. At a minimum, the software should work sufficiently to act as a demonstration to stakeholders of how and where things are going. Kanban does not provide this clear release point. With work items completed totally independently, we have to be able to recognize when it makes sense to turn a build into a release. One advantage of Scrum then, is that it allows for regular and meaningful stakeholder meetings which should, in turn, provide important stakeholder feedback, another important prerequisite for a project to say it is using itself as an Agile development process.
Inefficiencies in the Kanban process can arise if some workflow states have limits and some do not. The most obvious example is one in which the initial state is not limited. A whole lot of work will accumulate at the start and not be able to move forward. Despite there being a manageable amount of work actually being done, it would become difficult to inject change because technically, there is already a lot of work in progress. The WIP allowed in any one state is a function of the relative effort required to process a number of items through that state, the effort available, and the desired duration of that state per regulated work item. This could be calculated but a balance can also be achieved through regular measurement and process improvement.
To summarize, Scrum teams work together in the same fixed time slot called a sprint. They measure their velocity to control WIP for each subsequent Sprint. Kanban teams work more independently in variable-length states, limiting the work in each stage. They also measure lead-time, the time for any work item to make its way right through the system which could also be considered velocity, but to avoid confusion we just don't call it that in Kanban.
To offer a final analogy, Waterfall projects result in a dry faucet until suddenly a large mass of water gushes out all at once; Scrum delivers smaller amounts in regular spurts, and Kanban provides a steady trickle.
Change Friendliness
Because the content of a sprint should not be modified once it has begun, Scrum is less change-friendly than Kanban — which, by contrast, consumes one work element at a time so tasks or their priorities can be modified up until the last minute (whether it is wise to do so is another matter). Scrum requires sets of collected tasks to be defined in advance ready for upcoming sprints and while requirements can be added or removed from those un-started sprints, there is a knock-on effect as something else must either be removed or added to compensate. Further, with sprints often being themed, this also tends to make Scrum less change-friendly, as the change may not fit with the overall theme of the sprint. Kanban, having no themed iteration can manage a change without it being conceptually or practically disruptive.
The Task Board
When followed correctly, Scrum requires that tasks in the current sprint be visible to all team members on a 'task board'. This is often a whiteboard with sticky notes organized in groups representing the status of each task in the sprint. The board is usually a physical board, but can just as well be a virtual board in a software tool provided all team members have quick and easy access to it. Team members show their progress by moving the notes from one column to the next representing progress through the sprint. Once a sprint is complete, the task board is cleared ready for the set of tasks in the next sprint.
Without the equivalent of a short sprint, Kanban does not benefit from a task board in quite the same way. But it can still be valuable in presenting project status by having columns for each work state and sticky notes for tasks that are moved, not as they are completed in their current work state, but as they are accepted into the next state. In this way, the team can see whether tasks are getting backed up and adjust staffing or state limits to compensate. Unlike the Scrum board, the Kanban board persists throughout the project and is not cleared until the project ends.
Interestingly, it is acceptable for the Scrum task board to have columns for different work states within the sprint so that the team can get visibility of status at a greater level of granularity. As a result, it may look very much like a Kanban board, but of course, it is not.
The Burndown Chart
Because each sticky note on a whiteboard may represent a different-sized task and the task board does not show time, it is not always clear from the board what progress the team is making and whether they are on target to complete the sprint on time. A sprint burndown chart can provide this insight. A burndown chart, as its name suggests, shows the remaining effort on a vertical axis against time on the horizontal axis. It should be updated daily and, as with the task board, be easily visible to all. A release burndown chart can also be used. This is the same as the sprint burndown chart except that it shows effort remaining until a planned release. Having no fixed iterations, a sprint burndown chart in a Kanban project makes no sense. However, although not prescribed for Kanban, there is nothing to stop a Kanban project using a release burndown chart, which can certainly be useful.
Illustration 5: Burndown Chart for a release
The Scrum
Because all the players in a Scrum sprint are working to the same schedule, regular (daily is recommended) all-hands meetings are prescribed. These meetings (the scrum) should be short, stand-up, and held around the task board. Each team member describes his or her progress since the previous meeting, plans for that day, and any issues or requests for help. But don't wait for these meetings to ask for help — Agile projects are dynamic and immediate. In other words, if it needs doing, do it now.
By contrast, on any one day in a Kanban project, the varying states of work in progress can actually make daily team meetings disruptive. But don't disregard the quick, all-hands meeting concept entirely, just do it at a more appropriate frequency and only open the floor to new issues; not everybody needs to report everything they are doing.
Process Improvement
We have mentioned process improvement throughout this paper, and with good reason: it is a key aspect of any Agile project. Measure as much as possible and make corrections when it makes sense. Use the velocity of the team during the previous sprints to select an appropriate amount of work for the next sprint. But don't change the allocation of tasks in the middle of a sprint. If the sprint is running late, finish it anyway and make adjustment next time. With Scrum, accurate measurements can be taken as often as daily at the Scrum meetings.
When starting a new project using Kanban, it may not be obvious what the WIP should be for each work state. Make a best guess and if a bottleneck begins to form, either reduce the WIP before the bottleneck or increase the WIP right after the bottleneck, or both.
In both methods, it is a matter of trial and error, but herein lies one of the great benefits of Agile processes. Because they attack the problem in small increments, the process can be reviewed and improved after each increment, whether it be the velocity of a sprint or the time it took to complete a story point in Kanban. This is not true of the Waterfall model in which it is difficult to measure anything mid-project and realistically know what changes to make.
Maintenance
We have seen how Scrum methods require a time-bound sprint which, once started should not be disturbed. This makes it less attractive as a process for managing maintenance projects where tasks are primarily software bug fixes or other defect resolutions, as it would be difficult to accommodate an urgent fix without totally undermining the sprint. Estimating effort for a bug fix is also more difficult than for new work and so getting the right amount of work for a maintenance sprint is harder, leading to the possibility of more sprints failing to complete on time. Also, sprints are usually defined by some overall theme in which tasks can be designed, developed, and tested together. However, defect resolution is not a good theme and as discussed before, this can increase the areas of affected code and increase the risk of code being damaged by friendly fire.
By contrast, Kanban suffers from none of these disadvantages where maintenance is concerned and is therefore far better suited to maintenance. It is also far more accommodating of combined bug fixing and new work. It would not be a surprise to see a project originally run with Scrum principles, begin to take on Kanban principles as it moves into support and maintenance mode.
Summary
The overall conclusion to be drawn from comparing these two Agile methods is that Scrum is defined in more detail and provides far less flexibility. Consequently, Kanban serves as a very good introduction to Agile for traditional project teams, requiring less disruption and so, hopefully, less resistance (in fact, Kanban principles can be used within, and as part of, a Waterfall process!). As team members grow, adapt, and learn, projects can move toward a more Scrum-like discipline or remain flexible, whichever suits them most. Remember that even if a project does adopt Scrum as its process, as it transitions from development into maintenance it may be desirable to follow Kanban principles instead.
One final thought: for either of these Agile methods to work, there must be a paradigm shift in corporate expectation. While sales or business-driven commitments such as delivery content and delivery dates continue to be made, teams will struggle to meet them with the result that everyone will be disappointed. An Agile approach, while offering fewer commitments, can result in more success in delivering working functionality which keeps customers and, consequently, everybody else happy.