Taking the Red Pill
“You take the red pill, you stay in Wonderland, and I show you how deep the rabbit hole goes. Remember, all I’m offering is the truth. Nothing more.”
Morpheus, The Matrix (1999)
By Leonard Greski, Chief Scientist, LiminalArc
A summer conversation with a colleague about a potential assignment to help a client establish cause & effect between investments in projects and subsequent improvements in key performance indicators quickly morphed into a red pilling exercise about the sheer volume of research on corporate performance frameworks and measurements, as well as the usefulness of this research to practitioners.
An Amazon.com search for books on corporate performance measurement returns over 120 results. The same search on Google returns approximately 759,000 results. A 2024 review of the performance management literature, Evolving Landscape of Performance Measurement Research: A bibliometric analysis, identified 4,810 relevant papers in peer-reviewed journals from 780 journals, published between 1974 and 2020 (Rompho, et. al., 2024).
Clearly much has been written on this topic since The Balanced Scorecard – Measures that Drive Performance was published in the Harvard Business Review in 1992. One would think that 30+ years would be plenty of time to establish a theoretically sound, easy to implement mechanism to measure strategy execution, and that practitioners would adopt this mechanism to improve it. Yet a 2022 Gartner survey of 292 business leaders responsible for enterprise or IT business strategy indicates that less than half of the leaders surveyed (47%) believed their companies were fully executing their strategies (Gartner, 2023).
Unfortunately, my experience working with organizations ranging in size from startups to Fortune 10 behemoths indicates they frequently struggle with how to measure strategy execution, especially when it comes to establishing cause & effect between specific investments and expected improvements in business results. This article describes three challenges that are common to corporate performance measurement frameworks and provides guidance on how to overcome these challenges.
You Can’t “Unsee” the Naked Emperor
As of this writing, the most used measurement frameworks include the Balanced Scorecard, Objectives and Key Results (OKRs), and Key Performance Indicators (KPIs). Each measurement approach has its strengths and weaknesses, but collectively they suffer from a common set of challenges that prevent companies from obtaining valid, reliable and actionable results, including:
- Over-reliance on case studies that can’t be generalized across companies and industries,
- “The Devil’s in the Details,” and most importantly,
- Inability to establish cause end effect between actions taken and results.
Case Studies “Rock the World” (or not)
Business literature is replete with books and articles that rely heavily on case studies, an approach where the experience of a single firm, leader, department or team is described and held up as an example for other organizations to follow. The literature on corporate performance measurement is no exception to this pattern, such as Kaplan and Norton’s The Balanced Scorecard (1996), which includes a variety of case studies to illustrate key themes in the book. John Doerr’s book on OKRs, Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs even includes the case studies in its title.
While the case study approach makes for engaging storytelling and provides concrete “success stories,” its weaknesses are a function of its strengths. First, the approach tends to be biased towards success stories. After all, who wants to read about failed measurement framework implementations? Second, case studies are hard to generalize beyond a specific company because no two companies have identical sets of differentiating business capabilities (distinct combinations of people, business processes and assets that generate measurable value), even within the same industry.
A case in point can be taken directly from Measure What Matters, where Google’s Marissa Mayer was recognized as a “star pupil” with OKRs (Doerr, p. 7). Mayer served in several increasingly responsible roles at Google between 2005 and 2012, including membership in its operating committee. Her success at Google led to her being named as CEO of Yahoo! in 2012. However, Mayer was unable to replicate Google’s success with OKRs at Yahoo!, even though the two companies were in the same industry. If the author described “star pupil” can’t replicate her success with OKRs at a second company whose business model is very similar to the first company, clearly the OKR framework isn’t a primary driver of success.
Runnin’ with the Devil…
Another problem that is endemic across corporate performance measurement frameworks is that successful implementation of a framework is largely a combination of both the implementation details and the effectiveness of the communication about purpose and usage of the framework (Malina & Selto, 2001). Unfortunately, guidance about how to do this effectively is scant, at best, in the academic literature.
For example, a literature review of performance measurement systems (PMS) articles that were published between 2014 and 2020 (Owais & Kiss, 2020) found:
“…when explaining the relationship between PMSs and organizational outcomes it does not only depend on the presence or design of the PMS, but the degree of PMS sophistication, the type of PMS used, and the PMS characteristics also impact the extent to which objectives are realized.” (Owais & Kiss, 2020, p. 117).
The complexity and variation in implementation details across academic studies makes it very difficult for theoreticians to compare results, resulting in the conclusion that “The impact of these systems on organizational performance is vague, a positive association between organizational performance and strategic performance measurement systems is found in some studies, in contrast, other studies found that the relationship depends on organizational performance types” (Owais & Kiss, 2020, p. 112).
The vagueness of performance measurement systems’ impact on organizational performance leads one to question the risk / reward of investments in complex measurement and reporting capabilities because these systems are often expensive to implement and operate.
Does X Really Cause Y?
The last, and most important, challenge implementers of performance measurement systems encounter is establishing cause and effect between investments in an organization’s capabilities and business results. For example, the Balanced Scorecard model asserts that an organization’s financial performance is a function of customer outcomes, which is mediated by internal process effectiveness & efficiency and the organization’s capacity for improvement and innovation. The relationships between these concepts are assumed to be linear, as illustrated below.
Figure 1 Balanced Scorecard Causal Relationships
The conceptual model is then operationalized (i.e. made concrete) with by associating each of these perspectives with a set of directly observed variables, such as employee retention, process cycle times, cost per order, etc. An illustrative example including concrete measurements for each perspective looks like this:
Figure 2 Scorecard with Observed Variables
Without going into the details of the statistical techniques used to assess the validity or explanatory power of a model like this (e.g. structural equation modeling, confirmatory factor analysis, etc.), numerous researchers have asserted that the assumed logical and causal relationships between scorecard dimensions are recursive and dynamic (Neely, 2005), making the task of establishing cause and effect between investments in specific capabilities and their impact on financial performance exceedingly difficult.
That said, the Balanced Scorecard model supports the idea of articulating and aligning strategy across multiple levels in an organization, a process that the Balanced Scorecard Institute calls cascading (Balanced Scorecard Institute, 2019). As an organization cascades the scorecard from the overall organization to departments / business units, the decomposition of objectives can be used to more directly align specific investments with expected business outcomes, potentially mitigating the non-linearity problem. However, the cascade process brings with it a geometric expansion in the number of scorecards and organizational dependencies to be monitored by the organization.
Mtau and Rahul (2024) describe the need for a similar alignment process between an organization’s strategic objectives and Key Performance Indicators, and they assert that this process starts with an understanding of what is measured within an industry because it grounds an organization’s performance measures in readily available comparison benchmarks. However, a subsequent step in their process is Identify and Capture the Right Data, which is sufficiently ambiguous to be of little value in determining what to measure and how to measure it.
Separating the Wheat from the Chaff: what’s a business leader to do?
At this point we’ve painted a relatively bleak picture about the state of corporate performance measurement research over the past 30 years. Are performance measurement frameworks a scam, an overly ambitious theory that can’t effectively be implemented in practice? Yes, some of the claims about these frameworks are overhyped (e.g. OKRs don’t really “rock the world”). On the other hand, corporate performance measurement research has created a forum where researchers and business practitioners can improve our understanding about how non-financial aspects of business contribute to financial results.
After thirty years of research, the good news is that there are enough “kernels of wisdom” to be gleaned from the performance measurement literature and business practice that can be applied cross-industry to effectively measure strategy execution, including:
- Implement the simplest solution that makes sense for your business,
- Organize measurements around value streams and capabilities, and
- Establish cause and effect.