Introduction to Agile Software Development Methods

by Adam Sandman on Wednesday, August 21, 2024

Abstract

With all the Agile methods out there, how does someone with traditional waterfall or v-model experience begin to understand each? Which methods are truly Agile and which are not? What are the defining characteristics of each Agile method, as well as those not quite so Agile? What are the benefits of using each Agile method and what are some of the drawbacks? This article attempts to answer these questions in a concise and simple way and offer a launch pad for those who would like to know more.

Introduction

The Agile development movement began in earnest in the 1990s as a rejection of the establishment with its rather staid and seemingly sluggish development methods known generally by names such as the waterfall model or V-model. These older methods had gained a reputation for missing deadlines, going over budget or failing completely and Agile offered a means to make this a thing of the past. With most revolutions the ideas are rarely new and often promise more than they can actually deliver, and so it has proven with Agile methods. While there are considerable benefits to Agile methods, they are by no means a panacea for past ills and should only be adopted after serious consideration and careful planning. It is critical that the strengths and weaknesses of any method are understood before proceeding, and risks reduced through awareness and preparedness for potential pitfalls. This article aims to provide a quick starting point for understanding the most prominent methods, their individual characteristics and unique offerings.

All Agile methodologies share a set of core ideas which are prescribed from method to method in subtly different ways; iterative and incremental delivery of working code, frequent collaboration with stakeholders, closely working, self-organizing teams, and the ability to embrace change late in the project. Agile methods are shamelessly incestuous, borrowing from each other and using existing ideas in slightly different ways so it becomes very difficult to tell whether a project is following any one method as even the slightest adaptation of any aspect of a process can make it seem more like another.

Fortunately, the majority of the concepts are compatible so that it becomes very easy to use Agile methods like a smorgasbord, picking and choosing parts at will.

Figure 1: The iterative approach over the traditional approach

Some Agile proponents are so enthusiastic that they fail to recognize that Agile methods have drawbacks. They are not particularly adaptable to larger, enterprise or distributed developments where teams cannot all meet face-to-face and they are less well suited to fixed-price, contractual projects in which functionality is non-negotiable. They are also difficult to apply to embedded systems. Some Agile methods have little up-front design effort which can lead to considerable rework or integration problems. User involvement is also a double-edged sword when frequently changing ideas lead to more requirements churn than even Agile processes are prepared for.

The Agile Manifesto (www.agilemanifesto.org) provides a set of guidelines for those wishing to become more agile but it is not a set of rules and is does not in of itself define process. This leaves plenty of room for debate as to which methods are Agile and which are not. In this article we shall address those that are generally accepted as Agile followed by those for which argument continues. It should be stressed that while some methods describes themselves as Agile, or include ‘Agile’ in their name, that does not necessarily make them so.

Section 1. Yes, it is Agile!

Scrum

With prominent web sites such as www.scrum.org and www.scrumalliance.org to promote and support the Scrum method and those adopting it, Scrum has become one of the ‘go-to’ techniques for those striving to become Agile. Along with XP, it is perhaps the most well-known of the Agile methods, and also the most precisely defined which means that there is a lot of documentation and pre-built process for teams that are willing to adopt the methodology completely.

However, Scrum’s greatest strength is also its greatest weakness. With specifically described roles, short iterations, tight schedules, daily meetings and an insistence on a release-quality product after each iteration (or sprint in Scrum terms), Scrum can seem so very foreign to inexperienced teams that it can be difficult to achieve early success unless you have some in-house knowledge.

For those who are old-hands at the method however, it can prove quite effective. As with Agile methods generally, requirements begin as a simple prioritized list of needs with little detail, known as the backlog. Only enough of these needs are pulled from the backlog to fill a single sprint and to begin their short journey from refinement to tested software. A typical product backlog is shown below:

On their way, these needs will be elaborated by stakeholders, discussed in daily meetings (scrums), in which they are subjected to design proposals, used to create test cases, implemented, tested and delivered to stakeholders to review and provide input for subsequent sprints - and all this in just 2 to 4 weeks. Progress throughout the sprint is measured by the number of story points left to complete in that sprint and displayed using a ‘burndown chart’. A story point is an arbitrary measure of the complexity or effort required to implement a user story. The only requirement is that story points are consistently applied. The rate at which story points are completed is called the velocity. After each sprint there is a review and weaknesses in the process or team performance are identified and changes made. In this way the project should improve from one sprint to the next.

Figure 2: Typical burndown chart

The three most prominent roles in Scrum are the ScrumMaster, the Product Owner and the team member. The ScrumMaster acts as a super team leader, (but not a team member) keeping the team focused on the goals of the sprint, ensuring Agile principles continue to be followed, working with the Product Owner to keep the backlog up to date, and resolving any issues which might put the schedule at risk.

The Product Owner decides how to prioritize the requirements, making changes as necessary based on feedback from stakeholders. Note that the Product Owner does not choose which requirements go into a sprint exactly; that is dictated by the amount of work that can be done in the sprint and the backlog order. The Product Owner can only prioritize, not say how many items from the top of the list will be addressed next. A typical Sprint backlog is shown below:

Team members are those that design, code, test and produce the product. The team should consist of cross-functional individuals able turn their hands to whatever is required to meet the sprint goals, as well as at least one representative of the stakeholders. Completing the sprint on time and with all goals met is the responsibility of the team as a whole, it is not the job of the ScrumMaster!

While Scrum does not describe the whole process, what it does define it does so very specifically. To fill the gaps, most projects pull in ideas from other processes resulting in Agile projects that use a hybrid of more than one concept.

Extreme Programming (XP)

The idea developed by Kent Beck was to use best programming practices but take them to the extreme – hence the name. As such, none of the individual concepts of XP are really new, but their application and combination is what makes XP different, or at least did in the 2000s, which is when it initially became popular. XP advocates adaptability; recognizing that each project faces different problems, applying those methods that will work and discarding those that will not. Partial XP therefore becomes a true possibility, often combined with Scrum, which has gaps in its definition particularly suited to adoption of XP practices.

XP embraces standard Agile principles such as short iterations; requirements being loosely defined with user stories until they are needed; almost immediate and regular testing; close, on-site stakeholder involvement with a rapid feedback loop; and team responsibility for success or failure.

As is necessary for Agile projects, the XP team expects and accepts changes in requirements based on feedback from stakeholders and the obligatory high level of communication within the team. Intense communication is necessary because everyone is required to understand the system as a whole so that anyone can work on and change any of the code. This fact alone makes XP difficult (but not impossible) to apply to large projects. Quickly written code is often used as a means to explain ideas and try out designs. With short iterations comes regular layering of new functionality, thus refactoring is encouraged in order to consolidate separately written parts into a more efficient whole.

Testing taken to the extreme means using as many testing techniques as necessary, as often as possible. Methods include unit testing, acceptance testing, integration testing and test-first programming, (although this is debatably a coding technique not a testing technique.) Initially, daily integration testing was advocated, however this has sometimes proven impractical and the frequency practiced is often longer, perhaps weekly; a good example of taking from XP only that which works for a particular project.

Pair programming is part of XP, the idea being that two heads are better than one. Using only one workstation, one person writes code while the other observes, injecting ideas, pointing out potential problems and thinking about the ‘big picture’ as the code is written. Roles are reversed frequently and the pair work as if joined at the hip. If a pair get too comfortable with one another they may begin to think alike and two heads thinking as one is no better than one head, defeating the purpose. Pairs should be split up and reassigned when this happens. While pair programming does increase staffing needs, it is an excellent technique for building in quality which is an Agile ideal.

Finally: simplicity. The simplest solutions are encouraged, addressing the immediate problems, not problems that might arise tomorrow. This helps everyone to remain focused and meet deadlines but can result in considerable redesign and refactoring over time.

One of the reasons XP is so popular is its flexible nature. It is easy to combine features of XP with other ideas; indeed, XP is more about technique than process and so dovetails well with process-centric approaches such as that of Scrum.

Kanban

Kanban has the dubious distinction of being Agile without being iterative. Processes like Scrum have short iterations which mimic a project lifecycle on a small scale, having a distinct beginning and end for each iteration. Kanban requires that the software be developed in one large development cycle. Considering this, it is surprising to learn that Kanban is considered Agile at all, and yet it fulfils all twelve of the principles behind the Agile manifesto, because while it is not iterative, it is incremental.

The principle behind Kanban that allows it to be incremental and Agile, is limited throughput. With no iterations a Kanban project has no defined start or end points for individual work items; each can start and end independently from one another, and work items have no pre-determined duration for that matter. Instead, each phase of the lifecycle is recognized as having a limited capacity for work at any one time. A small work item is created from the prioritized and unstarted requirements list and then begins the development process, usually with some requirements elaboration. A work item is not allowed to move on to the next phase until some capacity opens up ahead. By controlling the number of tasks active at any one time, developers still approach the overall project incrementally which gives the opportunity for Agile principles to be applied. A typical Kanban board is illustrated below:

Kanban projects have Work In Progress (WIP) limits which are the measure of capacity that keeps the development team focused on only a small amount of work at one time. It is only as tasks are completed that new tasks are pulled into the cycle. WIP limits should be fine-tuned based on comparisons of expected versus actual effort for tasks that complete.

Kanban does not impose any role definition as say, Scrum does and along with the absence of formal iterations, role flexibility makes Kanban attractive to those who have been using waterfall-style development models and want to change but are afraid of the initial upheaval something like Scrum can cause while being adopted by a development team.

Section 2. It May be Agile!

DSDM/DSDM Atern (Incorporating RAD - Rapid Application Development)

Rapid Application Development (RAD) was championed by James Martin in his book of the same name in 1991, although the process had been around for some time before that. RAD is an iterative and incremental method which relies heavily on prototyping in order to obtain feedback from stakeholders. However, this requires early development of the GUI which can produce wasteful discarded versions and de-emphasize underlying functionality. RAD is not always considered an Agile approach and in 1994 the relative lack of discipline in the method led to the creation of the DSDM Consortium to develop a formalized independent RAD framework incorporating Agile principles. The result was the release of version 1 of the Dynamic Systems Development Method (DSDM) in1995 since when, it has continuously evolved leading to the launch of a new version in 2007 called DSDM Atern.

Figure 3: Features as variables in DSDM Atern

Like most Agile methods, DSDM Atern puts quality and schedule first, leaving functionality as the lone variable. The method of prioritization used is called MoSCoW, offering four simple requirements categories:

Must have;

Should have;

Could have; and

Won’t have this time around.

Figure 4: The relative sizes of pre and post-development phases varies between methods

Alongside all the standard principles that define Agile processes such as stakeholder involvement, and build early and often, DSDM Atern also advocates a degree of formal tracking and reporting which is not so common amongst Agile methods.

DSDM Atern addresses the narrow scope of some other methods such as Scrum by including pre and post-development phases in its purview making it a true project management process as opposed to a focused development process. It is also a method with a detailed process description and therefore it can take some time to embrace DSDM Atern fully.

Disciplined Agile Delivery (DAD)

To understand DAD well we need a little bit of history. In 1996 Rational Software Corporation created the Rational Unified Process (RUP) which was mostly a combination of UML and ideas developed by others such as Grady Booch and Jim Rumbaugh. RUP was a framework from which elements could be used for each project as necessary, which was a good job really as over the years it grew to be humongous and it is highly improbable that any single project used it all. The Agile Unified Process was developed in 2005 as a simplified version of RUP with work attributed to Scott Ambler who in 2012 wrote the book ‘Disciplined Agile Development’ with Mark Lines taking us from AUP to DAD.

A fair criticism of Scrum, XP and some other Agile methods is that they only address the software creation aspect of a project and leave other aspects, especially those at the start and end of the project, as a sort of ‘exercise for the reader.’ The DAD process framework attempts to expand the Agile influence beyond mere software generation to the entire product process while additionally addressing the needs of the enterprise such that the methods are scalable. As with its early forbears, DAD offers more than any single project would want and, in some cases, even proposes a number of alternative solutions from which to choose. Because DAD is rather comprehensive it is not easy to describe concisely in just a few paragraphs, but the following is a list of DADs characteristics summarized:

Self-organizing, cross functional teams of individuals with multiple skills often called generalizing specialists;

An environment that promotes learning to include user needs, how to improve processes and a growth in technical knowledge;

An adherence to the principles of the Agile manifesto;

The borrowing of ideas and principles from other Agile, iterative or lean methods such as XP, Kanban and Scrum;

An expansion of the vision from just software to all aspects of a product such as hardware and user process improvement and in doing so, extending agile principles to all project disciplines;

Encompassing the full project lifecycle from initiation to production delivery with an understanding that the nature of activities within iterations will change as the product matures;

Sufficient guidelines to help those starting up but not too many to put them into straightjackets, essentially trying to provide balance between too little and too much guidance;

Methods to address risk, evaluate product viability and ensure value through regular production of potentially deliverable solutions; and

Accommodation of demands from the enterprise such as governance, corporate vision, and other active projects teams.

Being relatively new, it remains to be seen whether DAD gains sufficient popularity to earn its place as a ‘standard’ alongside Scrum and XP. Even if it is not adopted in full, DAD offers ideas that can be integrated into other project environments.

Agile Unified Process (AUP)

The Agile Unified Process (AUP) was developed in 2005 as a simplified version of RUP with work attributed to Scott Ambler. It was subsequently updated in 2006 before Scott Ambler moved on to the work which became Disciplined Agile Delivery (DAD). RUP is process-heavy, and although the AUP is intended to conform to all the principles of the Agile Manifesto, it debatable how well it succeeds. For example, Agile methods should support self-organizing teams with no management hierarchy, however it is hard to ignore the fact that the AUP has, to some degree, inherited the process-heavy nature of its parent, RUP. The counter-argument is that the AUP does not enforce any of the process guidelines it offers. However, Project Management is still a major discipline in the AUP and if you begin to leave bits out to claim ‘agility’ are you really following that process?

Figure 5: Effort expended over the lifecycle for some of the AUP disciplines

As with some other Agile methods, initial requirements elicitation are excluded as is any delivery process at the back end. The Agile Unified Process is more up-front loaded than most Agile methods, requiring a considerable amount of modeling before implementation begins, which in turn demands some degree of early requirements analysis. While this is not a bad thing, care must be taken not to go too far and do waterfall-like detailed requirements analysis. Further, the proposed incremental development releases between production releases are not necessarily production quality and so again, the lifecycle can appear more waterfall than Agile.

Although the AUP is a good option for those trying to apply some Agile techniques to large-scale software projects, Scott Ambler’s relatively quick follow-up publication of the Disciplined Agile Delivery process in 2010 might be seen as an acknowledgement that the AUP was merely a stepping stone to something more viable.

Feature-driven Development

There are a number of factors which create some doubt as to whether Feature-driven Development is actually an Agile process, at least as it is defined by the Agile Manifesto. In FDD functionality is implemented iteratively and not according to business needs as might be decided by a Product Owner, but by functional area.

To see how requirements can be organized by functional area, they must be fairly well understood and so a design is created. From the design is derived a feature list at which point the iterative implementation cycles can begin.

Consequently, and as with some other methods, the first two phases of the FDD process are not iterative but take place just once which narrows the iterative part of the project, and so it could be argued that the agility of the overall project is reduced. At the same time there are some real benefits to this approach. For example, with each functional area written all at once there is less need to refactor than with a process where the same code might be repeatedly changed. Not surprisingly, less refactoring is one of the benefits of traditional development methods so in some sense FDD can claim to combine both traditional and Agile with some success.

Lean Software Development

Lean Software Development is less a process and more a set of principles to deliver by and consequently, the principles can be overlaid onto most processes that are truly Agile. For those familiar with Agile processes in general, some of the Lean philosophy seem very familiar. Reducing the time it takes for work to flow through the process cycle matches very well with Agile’s short iterations. Allowing teams to make decisions within acceptable constraints, and encouraging individuals to take the initiative are both Agile values. The elimination of unnecessary and wasteful work is also a value common to both Lean and Agile ideals.

Lean Software Development is gaining interest as a method in its own right but to date remains most widely found as a facet of projects using other process

Summary

While Agile methods have more similarities than differences, it is possible to pick out one or two distinguishing factors for each.

Scrum	Requires a ScrumMaster Burndown charts Limited to code production
XP	Pair programming Test-first programming Selectable ideas
Kanban	Incremental but not iterative Work In Progress (WIP) limits
DSDM Atern	MoSCoW prioritization Heavy in detail Addresses full lifecycle
DAD	Relatively new Addresses full lifecycle Borrows from elsewhere
AUP	Grew from RUP Heavy on pre-iterative phases
Feature Driven Development	Implementation by functional area
Lean Development	Ideas to overlay on other process