10: Development Methodologies
The author examines the two prevalent development methodologies (waterfall and agile) and suggests that agile development is a common "theme" in successful BU case studies.
Waterfall Development Process
"Waterfall" is described as the prevalent and preferred model for IT departments: there is a heavy period of requirements definition at the onset, then the development work is done in small increments to develop software over a period of "several months or years." Fundamentally, the waterfall process is: define the requirements precisely and build to specification.
This approach is reasonable for portions of a BI solution, particularly the back-end work of building the data warehouse, as the nature of this work focuses on laying a detailed plan and then delivering a fixed set of capabilities. However, it is not acceptable for business-facing solutions, which depend on greater flexibility, as they are based on a concept that must be refined by actual use to arrive at a useful solution and must have the ability to change and improve plans rather than being locked into a long development cycle.
The author "suspects that" the failure of early BI attempts were the result of projects that applied this approach, resulting in a team that defined requirements without analyzing the data and/or arrived at a plan that suited current needs, but that resulted in a system that was no longer relevant or useful to the business a year or two later, when the actual system was delivered.
The waterfall process is useful when the solution pertains to practices that remain largely the same, and components that are infrequently changed (such as purchased software packages and hardware). It is less useful when periodic change is needed (custom-coded processes, ETL processes, database structures) and counterproductive for things that must change frequently (reports, dashboards, calculation of performance indicators). BI resides in the latter domain.
BI also depends on flexibility, rather than rigidity, of requirements. A metric such as "success" is defined in a myriad of ways - different business units may need to consider different factors, and will constantly refine their definition in terms of the data considered and the calculations performed, and will need systems to adapt to keep pace. This is contrary to the waterfall approach, which would seek to establish a single, universal, and unchanging definition of success that can be most easily and accurately defined (based on system capabilities).
The implied conclusion is that traditional project management skills and methodologies, grounded in the waterfall approach, undermine the success of BI projects.
(EN: I think it would be more accurate to say that the traditional metrics for project management are counterproductive to success - as projects are deemed successful by measures of efficiency in regard to the variances of time, budget, and resource utilization - rather on whether the system produced is of any use whatsoever. Hence, a project team is provided incentives that pertain to specific tasks that do not relate to success for the organization, and which may lead them to work against organizational success to improve the efficiency of the project.)
Agile Development Techniques
The agile software development technique stands in stark contrast to waterfall: it is based on a broad set of requirements and the development of functional prototypes that can be delivered to the users rapidly (sometimes within a matter of days), who provide feedback that is used to refine the prototype.
Some of the key principles of agile development are:
- Rapid software development
- The ability to make corrections and changes rapidly
- Constant communication between users and developers
- Low cost and commitment that makes experimentation and discovery feasible
- Adaptability to changing needs
- Ability to capitalize on technological advancements
The agile methodology is also an iterative process: software development is not once-and-done, but build-and-improve. There is not a defined milestone at which a problem is considered completely solved, forever, nor are users left with the sense that the tools they are provided are fixed and immutable and that they must manage to do the best they can with what they are given.
This approach is viewed with some skepticism, as from the traditional waterfall perspective, it would seem to lead to a lot wasted effort developing throw-away applications that do not meet the user's stated needs, though the retort to this is that the traditional approach is no more fruitful: there is a great deal of churn in the requirements development process and, due to the length of time and amount of resources expended, often delivers applications that may meet the requirements, but do not meet the real needs of the user by the time they are delivered.
An added benefit of the agile method is that it is tested in the real world, buy real users. Instead of a small number of "experts" who attempt to discover or dictate what the users might require, agile involves the actual user, performing actual tasks, in a collaborative and incremental process of needs-definition.
Successful BI Project Management
The author discusses scope, resources, and time as the variables for managing BI projects successfully, and that any change in one of the variables affects the others, then goes into detail about their interaction (increased scope requires more time or resources, decreased timeline requires more resources or reduction in scope, decreased resources requires more time or a reduction in scope).
(EN: I don't see this as germane to BI, but to project management in general.)