jim.shamlin.com

Agile: An Executive Guide

by Jamie Lynn Cooke
IT Governance, 2011

Originally published as a booklet, there was not sufficient content for me to break it down into chapter notes, so it's listed as an article.

Information Technology (IT) projects have a reputation for missing deadlines, exceeding budgets, delivering little value, and thoroughly disappointing users. It is a reputation that has been earned, and are so commonplace that it has become an expectation when dealing with internal IT departments and vendors.

Aside of the short-term costs of executing poorly, a bad software project has an ongoing legacy: the fact that tech support has ballooned from a few part-time staff members to a fully staffed (and sizable) department is the direct result of bad solutions that require constant support.

The Agile methodology is not new, and has been around since the 1990s, and are intended to be a better approach to gathering user requirements and providing rapid development of solutions through a larger number of small-scale projects that are less expensive to rework or replace when their shortcomings are discovered.

The author assumes that he expects the reader of this book to be considering Agile - and his purpose is to provide information to help them make that decision, then to suggest a path to implementing it.

Chapter 1 - An Executive Brief on Agile

"Agile" is a term that refers to a collection of practices for software development that are geared toward rapid development. There is no single approach to Agile, as it embraces a broad range of methodologies that can be used. Instead, Agile focuses on specific objectives:

The author suggests Agile is presently in use by thousands of organizations worldwide, then provides a list of five (Nokia, Yahoo, Google, Microsoft, and BT).

A two-minute history of Agile

The author begins by considering the history of software development, beginning in the 1990s. (EN: this skips thirty or forty years of history, but let's take it as given). At that time, the IT industry was plagued by high failure rate in software development projects - substantial budget overruns, missed deadlines, bad solutions, and dissatisfied customers. Some in the industry attributed the problems to three key factors: too much planning, insufficient communication, and all-at-once delivery.

Technology projects traditionally begin with a long period of extensive documentation - requirements are gathered and documented as if they were legal documents, mountains of documentation is produced to describe the solution in minute detail. The purpose of generating such documentation was to give corporate managers a sense of confidence in what they were buying from their IT department, and the IT department the ability to prove it did what it promised, but the trade-offs are many:

Once these documents were created, the technical staff for delivering the solution would go away and develop "in a vacuum" - i.e., the developers referred to the documents and had no further contact with the business until the software was delivered and installed. The author again lists some of the consequences:

Finally, the software methodologies of the time led to all-at-once delivery, and a methodical process: analysis, design, development, testing, and delivery were done serially, requiring one step to be complete before the next would begin. Most significantly, nothing would be delivered to the user at all until the entire system was completely developed. Consequentially:

Various other factors such as limitations in technology or the lack of skilled resources are also cited for IT failures - but these are not within the control of the organization.

The core business benefits of Agile

The author lists ten benefits to using Agile:

  1. Ongoing risk management by regularly confirming and adjusting requirements with stakeholders who are constantly involved.
  2. Ongoing monitoring and adjustment of budget expenditures as the project progresses.
  3. Rapid delivery of tangible outcomes in regular releases.
  4. Ensuring features that drive the greatest business value are prioritized over those that deliver less
  5. Strong alignment with business requirements by including stakeholders in an ongoing review of software
  6. Responsiveness to business change by being able to redefine requirements while development is in progress
  7. Transparency of status tracking by including stakeholders in what was previously a "black box" process in the IT department.
  8. Substantially higher quality output by involving more individuals with greater diversity in the testing process
  9. Greater employee satisfaction by creating work environments in which team members are more interactive, more visible, and more often acknowledge for their contributions
  10. Minimized "whole-life" costs by incorporating usability and quality feedback into the development process to identify and avoid issues in production

Many of these values are delivered by greater involvement of others in the organization in software development activities - simply letting business people and developers interact with one another regularly during the development process.

Common Agile methodologies

Repeated: there is no single, definitive set of practices in agile, but there are a number of them that seem to be in widespread use because they dovetail with the principles of Agile.

Scrum

"Scrum" is an iterative project management method, commonly used in agile but not exclusive to it. Fundamentally, a larger project is broken down into individual pieces that can be accomplished in a two- to four-week "sprint." The business value of each piece is assessed, and they are worked in order, from highest to lowest value.

Each sprint begins with a planning meeting in which team members receive high-level requirements from the product owner, then agree to the items that will be delivered in the upcoming sprint. Each spring concludes with a "sprint review" at which the team demonstrates the work completed in the sprint as well as taking a retrospective of work processes to identify opportunities for improvement.

Scrum also involves daily standup meetings to discuss progress as well as a "dashboard" report that summarizes the work within the team. Other key documents include the project backlog (monitoring the progress toward the ultimate goal) as well as a spring backlog (an actual list of items for day-to-day work).

Dynamic Systems Development Method (DSDM)

DSDM places a strong emphasis on building prototypes to confirm the feasibility of a solution with the business prior to undertaking full development activities. It is primarily concerned with developing a range of artifacts (development plans, models) to be developed to gain confirmation from the business that the solution is aligned to their needs.

As with scrum, DSDM focuses on delivering software in time-boxed iterations and a collaborative working team, but develops prototypes and documentation prior to undertaking full development activities.

DSDM complements scrum by providing a structured framework for depicting the work that is proposed for a given sprint.

Feature-Driven Development (FDD)

FDD is an approach to software development that changes the focus on development to providing features (that deliver capabilities to users) rather than software functions (that perform programmatic tasks, but may be entirely invisible to the user).

FDD uses a "domain model" that describes a business problem and the proposed solution. Once identified, the model can be broken down into smaller unit that can be developed and delivered independently. Eventually, the features will be incorporated into the larger system - but initially, they are merely free-standing features.

In some respects, FDD meshes neatly with other Agile methodologies because it focuses on small pieces of work - but it is far more prescriptive about defining the boundaries of the solution in advance, assigning specific roles to specific individuals, and controlling the scope.

To some degree, its rigid nature can interfere with Agile's desire to be responsive to business change during the course of development.

Lean development

Lean development is an application of the Lean manufacturing process that originated in 1922. Lean seeks to identify and confirm the business value of various features, prioritize those that create the most value, and making the development process as efficient as possible.

Lean is also focused on identifying and eliminating areas of waste in the process - particularly where some members of the team are at a work stoppage because they are waiting on other members.

Lean also takes the position that decisions should not be made early in the process - but as late as possible, ideally just in time for the developers to begin coding. It also includes a feedback loop, such that the team receives regular critiques of their work and iterates for continuous improvement.

Another Agile-friendly quality is that Lean stresses the importance of having a cross-functional team with sufficient skills and authority to deliver high quality outcomes.

eXtreme Programming (XP)

The XP methodology focuses on delivering the simplest possible technical solution that meets the business objectives - recognizing that the greatest among of money and time is spent on deliberating over inessential and often inconsequential features. By considering high-level requirements and providing bare-bones solutions, XP delivers simple designs quickly and cheaply.

Another practice of XP is that is focuses on test-driven development (TDD) in which the first step is defining the tests that will be used in quality assurance, then developing systems that will pass the tests. This also tends to reduce feature bloat.

Refactoring is another practice that gives the team the authority to regularly review existing systems and modify it as required to facilitate future changes. The notion is that there is a short-term consumption of labor in order to gain a greater long-term benefit in terms of efficiency and flexibility.

Pair programming is another XP practice, in which programmers work in pairs (rather than individually) on the belief that this requires greater involvement and knowledge-sharing, as opposed to having one programmer develop code which others (lackadaisically) review.

Kanban

Kanban is a workload and change management methodology that can be used in conjunction with other Agile methodologies. It is most commonly applied to support and maintenance activities in which the workload is unpredictable and priorities shift constantly (work comes in and the queue is shuffled).

In order to have this flexibility, the technical team must organize their work into small tasks, with the understanding that tasks will be postponed when higher priority tasks arise. There is no commitment to a specific completion for any task, nor any commitment to handle a volume of work: the team commits to tasks they can actually deliver, no more and no less.

Kanban uses "boards" to disclose information about workflow: anyone can view the status of planned, current, and completed work; the ability to take on additional work; and an indication of impediments. This open communication and transparency promotes a culture of collaboration and trust.

While Kanban is valuable for managing tasks, it does not prescribe how the technical team should work with stakeholders during the development process or describe how features should be tested and integrated. As such, it is often used in conjunction with other Agile technologies.

Conclusion

The author reiterates that these methodologies are not the only ones that can be used in an Agile environment. There are several other common methodologies: links are provided in an appendix. (EN: Not preserving the names or the links - these methodologies come and go, and it would be better to do an online search to get the latest.)

Chapter 2 - What Can Agile Do For My Organization?

Agile methodologies are intended to manage the risk of change in software requirements that tend to arise due to the scope of development projects. In essence, breaking a large and long-term effort into shorter segments provides the ability to adapt and adjust as work progresses. As such, Agile offers few benefit to projects that are already modest in scope.

Agile is also geared toward solving the waste and efficiency that accumulates over time in larger and highly bureaucratic organizations. IT operations that are already effective and efficient would gain little from adopting Agile because they typically do not have the kinds of problems Agile is intended to solve.

Agile is also useful in managing the risk of resource availability and environmental changes that may occur in projects of a longer duration: resource availability, changing customer demands, a changing organization, and an unpredictable regulatory environment add risk to a long-term commitment to a set of requirements. The greater the risk of change, the more the need for a methodology that provides opportunity to adapt.

Ultimately, Agile is a solution to a specific set of problems: and firms that do not have those problems will benefit little from adopting Agile. They may leverage some Agile methods to achieve some benefits or address some issues - but the results will be less dramatic, and in some instances almost negligible.

Dispelling Agile myths

The author wishes to address some of the common misconceptions about agile.

One misconception is that Agile is a "quick fix" that will effect an instant turnaround. It is a significant change, and it will take considerable time for staff to fully transition to Agile. There will be change-management and training costs up front, and it will take time to be utilized widely enough to result in appreciable financial benefits.

Another misconception is that Agile projects require no planning - the staff can jump in and begin work immediately because the word is ad-hoc. However, agile does not do away with the need to plan, it merely makes planning an incremental task, done with each iteration rather than all at once before work begins.

Another misconception is that agile projects do not require documentation. However, being iterative is not the same as being ad-hoc. Organizations will still rely on formal documentation for compliance purposes, as a record of communication, as a reference for future development, etc. The software does not speak for itself - and in fact, some Agile methodologies include documentation as a deliverable. The difference is that Agile focuses on delivering functionality first and considers documentation to be a separate concern (the documentation need not be perfect in every detail before coding can begin) - the documentation remains a necessity.

Another misconception: Agile projects are not managed. While Agile does not yield to the highly bureaucratic management style of many organization, and delegates a great deal of decision-making authority to the team, it is not entirely autonomous or unsupervised. A quote from General Patton: "Never tell people how to do things. Tell them what to do and they will surprise you with their ingenuity." In that nature, management gives specific objectives to an Agile team, but does not micromanage their day-to-day work. This better leverages the skills and capabilities of staff, and enables management to minimize the amount of time spent (wasted) on details that should be beneath their level of concern.

Finally, Agile is often perceived as being too risky or radical for an organization. However, the reason that drives companies to consider Agile is to reduce the risk inherent in the existing processes. The concerns about its being "radical" merely reflect a level of discomfort with making a change from a familiar process that is not working very well in the first place.

Your Agile ROI

Calculating the potential for retrun on investment is contingent on four main factors:

  1. The business value that IT generates
  2. Current costs to develop and implement solutions
  3. Current cost to support, maintain, and update systems
  4. Costs related to staff turnover (hiring, training, loss of experience)

Other factors may exist, but these four seem to be the most directly related and quantifiable costs that Agile can reduce. Suggestions are provided for some of the ways in which Agile can be linked to quantifiable benefits:

It's also important to concede that Agile methodology is not likely to have a dramatic and immediate impact, and will incur costs in the short run before delivering returns gradually over a longer period of time.

With this in mind, a side-by-side comparison of development costs dos not tell the full story of Agile's potential financial benefits: it should also consider the whole-life cost of a solution - including the costs to maintain and support solutions, train users and manage change control, and other expenses related to the long-term benefits.

Your organizational culture

Culture is another element in determining whether a given methodology is a good fir for a given organization. In that regard, the author suggest three critical questions:

A "no" answer to these questions means Agile will be a challenge to implement because it would require a significant change in the organizational culture.

At the heart of Agile is the firm believe that people can, and will, do the right thing when given the opportunity. If top management sees employees as unmotivated people who must be closely supervised to get any work done, they will be unwilling to trust self-directed teams. And ironically, it is often lack of management trust that breeds an unmotivated staff.

Agile methodologies rely on an environment of empowered employees with mutual trust between stakeholders and the technical team members. The stakeholders, who are typically in positions of authority, must refrain from micromanagement: provide direction to the team, monitor to ensure their efforts remain on track, but otherwise refrain from interfering in their day-to-day work.

Equally important is the establishment of trust and mutual respect among peers on technical teams: effective teamwork can only be achieved in a collaborative setting where employees work together to find solutions rather than dividing up tasks and engaging in "finger pointing" when the work is not going smoothly.

The "business as usual" mindset is often an obstacle to realizing the benefits of Agile, as firms attempt to adapt certain methodologies and abandon others in order to continue to work as they have in the past. This will take employees out of their comfort zone of established working patterns, and the change will be difficult for some to tolerate.

The cultural barriers to adopting Agile can be significant, but are not insurmountable. More details on change management and making a graceful transition will be provided in the third chapter.

Your decision

The author does a bit more cheerleading: Agile is a radical shift for many organizations, but major improvements are seldom effected without making major changes. It is not sufficient to make a decision to adopt Agile, but it will need ongoing support in order to take root.

One of the most compelling arguments in favor of adopting Agile on a trial basis is that there are little start-up costs and it can be attempted on a small scale (one team, one project) without significant impact to the rest of the organization, then scaled up slowly once its benefits are proven out.

(EN: I don't entirely agree. In some situations, working in an Agile fashion requires setting up a separate environment, both in terms of systems and the physical space, to prevent Agile teams from being entangled by or interfering with traditional operations. If a project is chosen carefully, to mitigate dependencies, it can be cheap - but it is not necessarily so.)

Chapter 3 - Five Steps to Agile Success

The author outlines a five-step process to implementing Agile:

  1. Choose the right kickoff point
  2. Avoid common traps
  3. Establish a baseline
  4. Monitor the investment
  5. Expand Agile

(EN: This doesn't seem to be a five step process, each proceeding in order, but instead a list of items to consider in more-or-less chronological order.)

Choose the right kick-off point

Even though Agile has great potential to make a dramatic change, it would be a mistake to dive in all at once, heedless of the constraints and challenges of your organization. It may be possible to do so, but more advisable to seek a safe trial of the methodology before implementing it more broadly. There aer a few, rare examples where this has been done.

An ideal candidate for a trial project would be a new development project that involved a self-contained system that would not be burdened by the constraints of legacy systems or existing bureaucratic processes. At the same time, if the outcome is to have influence, the project must be important and visible enough that its success will be noticeable to the organization.

It may also be useful to find a group of individuals within the organization that are more adaptable and flexible in regard to change: a new team, with a new manager, is less likely to have become institutionalized.

Once the project and team have been selected, the next step is to choose among the available methodologies that support Agile. There is no one set of practices that you are required to use, and the nature of Agile is to do what makes sense and be prepared to switch if the chosen methodologies are not achieving results. Nor is it productive to use the same methodologies on every project. Especially for a trial run, the team should be prepared, and empowered, to change methodologies on the fly.

Some of the factors to consider in selecting methodologies include:

The author mentions the "Agile by Stealth" approach in which technical team members have decided among themselves, without the support or awareness of management, to leverage Agile methodologies as a necessary way to "get things done under the radar," meanwhile going through the typical rituals as to avoid provoking the ire of management. In some instances, finding teams that have been secretly using agile may be useful in suggesting it can be used overtly, but they are unlikely to be able to provide ROI metrics.

(EN: This strikes me as potentially dangerous in certain operations, where dedication to process could instead result in a backlash up to and including termination of employees who were failing to follow documented procedures. Proceed with caution.)

Avoid common traps

The simplicity that makes Agile so appealing also makes it highly susceptible to misapplication and misuse. In many instances, it is said that Agile has failed when an organization has failed to implement it properly.

One common problem with Agile is that its methodologies are adapted to suit existing work practices - for example, the project management model is run in an iteration-based fashion but the project planning is still done up-front with a detailed and approved plan that is not open to change or adaptation.

Similarly dangerous is applying Agile practices without understanding the rational for them. For example, holding daily stand-up meetings to report progress without the participation of key stakeholders who can resolve conflicts and remove obstacles yields little value (people report each day that they are still impeded by the same problems as the day before).

Insufficient communication and/or training is the root of many problems: participants are put into an Agile environment and expected to figure things out as they go along, and as such fall back to more familiar working patterns or misapply some of the practices of Agile.

Finally, Agile is often regarded as a philosophy rather than a tool, particularly when staff do not recognize the way that they fit within the organization. Aside of the schism between what works in theory versus practice, an enforced philosophy prevents adaptation - i.e., people are prevented from doing what works because they are following a rigid process. This is precisely what Agile is intended to correct.

Establish a baseline

In order to establish an ROI for agile, a firm must currently assess and record the net business value of equivalent work procedures. Otherwise, your estimation of the benefits of Agile may be dismissed as an apples-to-oranges comparison.

The author identifies the four key components of ROI:

  1. The business value that is generated by software solutions
  2. The current costs to develop software solutions
  3. The current costs to support, maintain, and update software solutions
  4. The current costs in training staff and managing turnover

A list of key metrics is also provided:

Qualitative metrics should also be considered:

Data of this kind is likely already available in high-level budget reports, incremental project status reports, sales figures and surveys, issue registers and support logs, and anecdotal reports of success or failure of software projects.

The purest ROI comparison scenario would compare two identical projects, which have the same objectives, the same budget, the same number of staff, the same time frame, etc. This would minimize the influence of factors other than development methodology. However, it's highly unlikely ever to occur - you will likely need to settle for similar projects and adjust the figures accordingly.

If you are not confident in the comparison, your options are to proceed with partial information, to track Agile without a benchmark for comparison, or to delay implementation until you can gather metrics on an existing project.

Monitor the investment

Agile methodologies provide a number of mechanisms for tracking progress, including formal reports, status update tools, and ongoing communications with stakeholders. The most valuable measure may be the delivery of outputs - i.e., the functionality delivered to the business at the end of each iteration, which begins delivering business value as soon as it is released.

One Agile teams that use four-week iterations, this can be closely timed to coincide with monthly reports, enabling the team to present an analysis in standard reports that already have an established audience.

Measuring the value of Agile against other methodologies is, on the surface, a straightforward comparison. However, you must normalize the variables to account for discrepancies - and this can pose significant challenges when comparing traditional software development to Agile.

Another challenge, independent of methodology, is in comparing the costs of an in-flight project to a completed one, as many costs (such as the ongoing cost of maintenance) are unknown for an ongoing project and will not be known until the software has been released and results can be measured.

As a cautionary note: the most advantageous comparison between Agile and other methodologies is not the side-by-side comparison of software development costs, but a measure of the whole-life costs that also includes communication, training, maintenance, and support.

ROI is not limited to the comparison of Agile to non-Agile projects, but should also be used across Agile projects to identify best practices and opportunities for improvement. It will be easier to make an apples-to-apples comparison between projects that are similar in structure. The value in doing so is to help refine which methodologies are most productive. (EN: This should come with the caveat that this seems to point to standardizing practices rather than choosing the right tool for the job.)

Another caution: for early Agile projects, the amount of training and overhead will be exaggerated; but when the team settles into Agile these costs will normalize, and the cost to onboard new teams will likewise be reduced by their ability to draw on internal knowledge.

Finally, although this current section has been focused on cost-and-benefit metrics, the author cautions readers about getting too caught up in the "numbers game." While Agile can reduce the whole-life cost of software, its chief advantages are qualitative: ensuring software is aligned with business needs, delivering value quickly, decreasing frustration, improving responsiveness, etc. Many of the benefits of agile will be evident in the things you will no longer have to do, or do less often: emergency fixes, work-arounds, and the like that arise because of deficiencies and poor efficiencies of other development methodologies.

Expand Agile

As your organization proceeds through a trail run of Agile, be attentive to whether it is delivering the desired benefits. If all goes well, you should see that work is being done more efficiency, stakeholders are more satisfied with their outcomes, employees have higher morale, and the quality of work is increasing.

The success you achieve with Agile should provide the details you need to encourage a broader adoption. Of course, the converse is possible: you may in fact discover that Agile methodologies are not suited to your organizational culture and should not be extended further.

The author suggests four key elements in a strategy to broaden awareness and adoption of Agile:

  1. Educating the organization on the qualitative and quantitative benefits of Agile: make presentations, publicize it on the intranet, and report outcomes
  2. Motivating people in the organization to establish a trial project in their own areas for testing the Agile methodology
  3. Identifying interested areas for which Agile is a good fit
  4. Collaborating with those who experiment with Agile to help them learn best methods from other internal trials.

As the adoption of Agile grows and matures, it is also advisable to create a community of practice and enlist qualified internal consultants to help it along.

Each time employees work on an Agile project, it's likely that the organization will gain confidence in the methodology. Management's initial reaction to an untried methodology is "no," but their reaction to a methodology that produces results is "more." (EN: This is not necessarily a good thing, when "more" leads a promising new methodology to be overextended and misapplied; it does more harm than good.)

Appendix - More Information About Agile

(EN: The author provides a rather lengthy list of links to additional resources, some of which are already defunct. I've decided not to curate or annotate them as a result. It's likely a Google search will turn up active and up-to-date information.)

JIM.SHAMLIN.COM
DISCLAIMERS · CONTACT