4: Measures of Success
There is not a universal "yardstick" for measuring the success of BI, though it's noted that most "successful" implementations do not use ROI. She asserts that vendors and consultants claim there is a high BI failure rate, and suspects their motives for doing so is that they offer a "fix" to these failures. In her own survey, about two-thirds rated their BI as being "moderately successful" and a quarter as "very successful," with less than 8% deeming their deployments as "a failure."
These ratings generally corresponded to the degree to which respondents felt that the Bi systems "contribute to company performance," but concedes that this is likewise vague as there are various ways to measure performance and the questionnaire did not ask respondents to indicate which aspects of performance were most impacted by BI.
Return on Investment
While the cost of a BI project is simple to estimate, the "return" that results is difficult to measure, as the effect of BI is indirect. Clearly, the profitability of a firm is impacted by the quality of the decisions made by employees on varying levels, and BI is a tool that provides information for decision-making purposes, but it would be difficult to find a solid causal connection between a profitable decision and the tools that provided information into the decision-making process.
It's also worth noting that ROI calculations are problematic in general. For example, the benefit upgrade that improves the stability of a server has the benefit of preventing downtime, but one cannot do an ROI calculation based on the amount of loss to the business that would have occurred had the upgrade not been installed.
At best, the "value" of a BI system can be ascribed to the decisions made throughout the business by those who utilize the system - though even this is a tenuous argument, in that there is no way to prove that a decision was made because of BI, or that the same decision would not have been made in its absence.
The author provides anecdotal evidence from a company that deems its BI deployment as successful, and which views BI as an infrastructure cost - having business intelligence about operations is just "the cost of doing business" and a factor in the firm's competitive posture.
Number of Users
Another metric for BI success is the number of users of the system, though this is also an area of much speculation and potential exaggeration. For example, if information from a BI system is printed in a newsletter, is every recipient of the newsletter to be counted as a BI user? If the information enhances their productivity or enables them to make better decisions, it is reasonable to conclude that they have benefited from using the information provided by the system. On the other hand, since newsletter recipients were unable to access the system to perform or contribute to the analysis, so some would not count them as BI users.
As a result, the penetration figures (which measure the percentage of employees who use the BI systems) vary widely from firm to firm, and there is no standard definition for comparison. The author's survey inquired as to the percentage of total employees that have access to a BI tool, and the average was about 25%
Access to the tools does not mean that they are actively using them, and the author notes that, among survey respondents, many made the distinction between "active" users and "defined" users
There is also the question of whether all employees should have access to BI tools, as in most instances, the activities of the rank-and-file workforce does not involve the consideration of enterprise data. Survey respondents indicated that they thought that about 55% of employees should have access to a BI tool.
However, this figure varied by segment. Most respondents felt that management and executives needed access to BI, and those in information-intensive areas (information systems, finance, research, and the like) remarked that a higher percentage of their employees could benefit from BI tools.
The author thinks that this may be a matter of perception, and mentions that managers once felt that most workers wouldn't need to have access to a computer (and it is now an indispensable tool), and predicts that as companies make BI relevant to their operations, an increasing percentage of workers will rely upon business intelligence to support their daily decisions and actions.
Other Measures of Success
The author provides a number of other measurements for BI success:
- The number of BI applications (dashboards, business views, etc.)
- The number of new requests placed in the system (as an indication of how much the system is being utilized)
- The number of standard reports and ad-hoc queries run by the system
- The elimination of independent databases and spreadsheets for information gathering
- Increased employee satisfaction (which implies that employers who are given the proper tools to do their jobs are more successful and "feel better" about their work)
- Increased customer satisfaction (another indirect measure that presumes that the quality of customer service is impacted by sound decision-making based on close attention to customer behavior)
The Role of Time
It's noted that BI deployments are sometimes an immediate success, but it is more likely that it may take years for the effect to be noticeable, given that BI is a major undertaking and requires a change in corporate culture and enterprise-wide practices. The author notes a correspondence between the length of time that companies have been using BI systems and the companies' perception of the success of those systems.