IT for Analytical Competition

Technologists use the term business intelligence (often shortened as BI) to encompass not only analytics–the use of data to analyze, forecast, predict, optimize, and so on–but also the processes and technologies used for collecting, managing, and reporting decision-oriented data. BI usually consists of two major types of activities: reporting and analytics. Reporting is desirable and important, but analytics–the use of sophisticated quantitative tools to model, predict, and optimize business processes–can provide substantial competitive advantage.

While analytical activities have long been important and provided value, they were often marginal to the mainstream of the business. Analytics were not visible to senior executives, were not central to the business, and were not part of the competitive strategy. What is new is that for an increasing number of companies, these activities have moved from the margins to the mainstream. For many, the use of analytics has become a primary activity used to support the overall business strategy.

The aspects of a business that can be optimized through analytics vary based on its strategy. For one business it may be marketing, by finding the best customers and charging the right price. For another, the use of analytics might be most important in optimizing the supply chain by minimizing inventory and maximizing availability.

Competing on analytics isn’t only about information and technology (there is a substantial human component), but those are obviously critical resources. IT architects and CIOs must determine how the IT infrastructure (hardware, software, and networks) will work together to provide the data, technology, and support needed by the business. This task is easier for recent start-ups such as Netflix that can create their IT environment with analytical competition in mind from the outset. In large established organizations, however, the IT infrastructure can sometimes inhibit serious analytical initiatives. Most IT environments were optimized for transactional applications rather than analytical ones.

At the beginning stages of analytical competition, companies may need to address problems of missing or poor-quality data, multiple definitions of its data, and poorly integrated systems. This is a key prerequisite for serious analytical work Another early-stage issue may be that the organization collects transaction data efficiently but often lacks the right data for better decision making. Even organizations that have long pursued some forms of analytical activity may have a poorly-integrated proliferation of business intelligence tools and data marts.

Analytical Nirvana
As firms approach full-bore analytical competition, however, they have typically put a variety of architectural tools and frameworks in place. By this time the organization has high-quality data, an enterprise-wide analytical plan, IT processes and governance principles, and some embedded or automated analytics. The most aggressive firms will have a full-fledged analytical architecture that is enterprise-wide, often automated and integrated into processes, and capable of real-time analyses where that level of responsiveness is necessary. Other indications of a high level of analytical capability include:

• Analysts have direct, nearly instantaneous access to data.
   
• Information workers spend their time analyzing data and understanding its implications rather than collecting and formatting data.
   
• Managers focus on improving processes and business performance, not culling data from laptops, reports, and transaction systems.
   
• Managers never argue over whose numbers are accurate.
   
• Data is managed from an enterprise-wide perspective throughout its life cycle, from its initial creation to archiving or destruction.
   
• A hypothesis can be quickly analyzed and tested without a lot of manual behind-the-scenes preparation beforehand.
   
• Both the supply and demand sides of the business rely on forecasts that are aligned and have been developed using a consistent set of data.
   
• High-volume, mission-critical, decision-making processes are highly automated and integrated.
   
• Data and analyses are routinely and automatically shared between the company and its customers and suppliers.
   
• Reports and analyses seamlessly integrate and synthesize information from many sources.
   
• Companies manage data as a strategic corporate resource in all business initiatives.

Conceptually, it’s useful to break the business intelligence architecture into the six elements described in Figure 1.

We’ll look at each element in turn with reference to analytical competition, with particular attention to data since it drives all the other architectural decisions.

Data Management
The goal of well-designed data management strategy is to ensure that the organization has the right information and uses it appropriately. Systems for enterprise resource planning, customer relationship management, and point-of-sale transactions, among others, ensure that no transaction or exchange occurs without leaving a mark. Many organizations also purchase externally gathered data from syndicated providers such as IRI and Nielsen in consumer products and IMS Health in pharmaceuticals.

The greatest data challenge facing companies is “dirty” data: information that is inconsistent, fragmented, and out of context. Even the best companies often struggle to address their data issues. We found that companies that compete on analytics devote extraordinary attention to data management processes and governance. Capital One, for example, estimates that 25 percent of its IT organization works on data issues–an unusually high percentage compared to other firms.

There’s a significant payoff for those who invest the effort to master data management. For example, Continental Airlines integrates 10 terabytes of data from 25 operational systems into its data warehouse. The data is used in analytical applications for both real-time alerts and long-range strategic analysis. Alerts notify customer agents of delays in incoming flights and identify incoming frequent-flyer customers who are assigned alternative flights if they are unlikely to make their connections. Marketing analysts use other data collected by the systems to study customer and pricing trends, and logistical analysts plan the optimal positioning of planes and crews. The company estimates that it has saved more than $250 million in the first five years of its data warehousing and business intelligence activities–representing an ROI of over 1,000 percent.1

Transformation Tools and Processes
For data to become usable by managers, it must first go through a process known as ETL, for extract, transform, and load. While extracting data from its source and loading it into a repository are fairly straight-forward tasks, cleaning and transforming data is a bigger issue.

In order to make the data in a warehouse decision ready, it is necessary to first clean and validate it using business rules and data cleansing tools such as Data-Flux and Trillium. (For example, a simple rule might be to have a full 9 digit ZIP code for all US addresses.) Transformation procedures define the business logic that maps data from its source to its destination. Both business and IT managers must expend significant effort in order to transform data into usable information. While automated tools can ease this process, considerable manual effort is still required. Informatica’s CEO Sohaib Abbasi estimates that “for every dollar spent on integration technology, around seven to eight dollars is spent on labor [for manual data coding].”2

Transformation also entails standardizing data definitions to make certain that business concepts have consistent, comparable definitions across the organization For example, a “customer” may be defined as a company in one system, but as an individual placing an order in another. It also requires managers to decide what to do about data that is missing. These mundane but critical tasks require an ongoing effort, because new issues seem to constantly arise.

Repositories
Organizations have several options for organizing and storing their analytical data:

• Data warehouses are databases that contain integrated data from different sources and are regularly updated. A data warehouse may be a module of an enterprise system or an independent database. Some companies also employ a staging database that is used to get data from many different sources ready for the data warehouse.
   
• A data mart can refer to a separate repository or to a partitioned section of the overall data warehouse. Data marts are generally used to support a single business function or process and usually contain some predetermined analyses so that managers can independently slice and dice some data without having statistical expertise. This approach is rarely used today, as it results in balkanization of data and creates maintenance problems for the IT department.
   
• A metadata repository contains technical information and a data definition, including information about the source, how it is calculated, bibliographic information, and the unit of measurement. It may include information about data reliability, accuracy, and instructions on how the data should be applied. A common metadata repository used by all analytical applications is critical to ensure data consistency.

Once the data is organized and ready, it is time to determine the analytic technologies and applications needed.

Analytical Tools and Aplications
Choosing the right software tools or applications for a given decision depends on several factors. The first task is to determine how thoroughly decision making should be embedded into business processes. Should there be a human who reviews the data and analytics and makes a decision, or should the decision be automated and something that happens in the natural process workflow? If the answer is the latter, there are technologies that both structure the workflow and provide decision rules–either quantitative or qualitative–to make the decision.

The next decision is whether to use a third-party application or create a custom solution. A growing number of functionally or industry-specific business applications, such as capital-budgeting or mortgage-pricing models, now exist. Enterprise systems vendors such as Oracle and SAP are building more (and more sophisticated) analytical applications into their products. According to IDC, projects that implement a packaged analytical application yield a median ROI of 140 percent, while custom development using analytical tools yields a median ROI of 104 percent. The “make or buy” decision hinges upon whether a packaged solution exists and whether the level of skill required exists within the organization.3

But there are also many powerful tools for data analysis that allow organizations to develop their own analyses. Major vendors such as Business Objects and SAS offer product suites consisting of integrated tools and applications. Some tools are designed to drill down to predetermined views of the data, while others are more statistically sophisticated. Some tools can accommodate a variety of data types, while others are more limited (to highly structured data or textual analysis, for example).

Whether a custom solution or off-the-shelf application is used, the business IT organization must accommodate a variety of tools for different types of data analysis. Employees naturally tend to prefer familiar products such as a spreadsheet, even if is ill-suited for the analysis to be done. But without an overall architecture to guide tool selection, excessive technological proliferation can result. In a 2005 survey, respondents from large organizations reported that their organizations averaged 13 business intelligence tools from an average of 3.2 vendors.4 In the past, this was probably necessary, as different vendors had different capabilities–one might focus on financial reporting, another on ad-hoc query, and yet another on statistical analysis. While there is still variation among vendors, the leading providers have begun to offer business intelligence suites with stronger, more integrated capabilities.

Presentation Tools and Aplications
Since an analysis is only valuable if it is acted upon, analytic competitors must empower their people to impart their insights to others through reporting tools, scorecards, and portals. Presentation tools should allow users to create ad hoc reports, to interactively visualize complex data, to be alerted to exceptions through a variety of communication tools (such as email, PDAs, or pagers), and to collaboratively share data. Business intelligence vendors such as Business Objects, Cognos, SAS, and Hyperion sell product suites that include data presentation and reporting solutions. As enterprise systems become more analytical, vendors such as SAP and Oracle are rapidly incorporating these capabilities as well. A new generation of visual analytical tools–from new vendors such as Spotfire and Visual Sciences and from traditional analytics providers such as SAS–allows the manipulation of data and analyses through an intuitive visual interface.

Operational Processes
This element of the analytical architecture answers questions about how the organization creates, manages, and maintains data and applications. It details how a standard set of approved tools and technologies are used to ensure the reliability, scalability and security of the IT environment. Standards, policies and processes must also be defined and enforced across the entire organization.

Issues such as privacy and security, as well as the ability to archive and audit the data are of critical importance to ensure the integrity of the data. This is a business as well as a technical concern, as lapses in privacy and security (for example, if customer credit card data is stolen) can have dire consequences. It should get executives’ attention that they can be found criminally negligent if they fail to establish procedures to document and demonstrate the validity of data used for business decisions.

Conclusion
For most organizations, an enterprise-wide approach to managing data and analytics will be a major departure from current practice. Top management can help the IT architecture team to plan a robust technical environment by establishing guiding principles for analytical architecture. Those principles can help to ensure that architectural decisions are aligned with business strategy, corporate culture, and management style.5 To make that happen, senior management must be committed to the process. Working with IT, senior managers must establish and rigorously enforce comprehensive data management policies, including data standards and consistency in data definitions. They must be committed to the creation and use of high-quality data that is scalable, integrated, well documented, consistent, and standardsbased. And they must emphasize that the analytical architecture should be flexible and able to adapt to changing business needs and objectives. A rigid architecture for analytics won’t serve the needs of the business in a fast-changing environment.

Endnotes
1. Hugh Watson, Barbara Wixom, et al., “Real Time Business Intelligence: Best Practices at Continental Airlines,” Information Systems Management, Winter 2006, 7-18.
2. Madan Sheina, “Refocused: Back to the Future,” Computer Business Review, September 2005.
3. Henry Morris et al., The Financial Impact of Business Analytics: Distribution of Results by ROI Category, International Data Corporation, IDC # 28689, January 2003.
4. Enterprise Business Intelligence: Strategies and Technologies for Deploying BI on an Enterprise Scale, Data Warehousing Institute, August 2005.
5. The principles-based approach to IT architecture is described in Thomas H. Davenport, Michael Hammer, and Tauno Metsisto, “How Executives Can Shape Their Company’s Information Systems,” Harvard Business Review, (March-April 1989) 130-134.

Thomas H. Davenport is a Fellow with the Accenture Institute for High Performance Business and holds the President’s Chair in Information Technology and Management at Babson College. He is a widely published author and acclaimed speaker on the topics of information and knowledge management, reengineering, enterprise systems, and electronic business and markets. He has a Ph.D. from Harvard University in organizational behavior and has taught at the Harvard Business School, the University of Chicago, Dartmouth’s Tuck School of Business, and the University of Texas at Austin. He has also directed research centers at Ernst & Young, McKinsey & Company, and CSC Index.