Developing an Estimation Framework in IT

Most people would accept that IT’s track record in estimation is not great. So how can we improve estimation? To improve estimation we need to understand the challenges and life cycle of estimates over the course of a project and what information estimates need to be based on.

While this article touches on aspects of some related processes, e.g., requirements capture and design, it doesn’t aim to focus on these processes per se. Instead, it outlines some of the issues encountered in the development of an estimation framework to support the governance of complex IT projects and programs.

What is Estimation and Why is it Important?

Estimates are needed so that decisions can be made. They enable objective decision making. They provide a predicted value or, what is often more useful, a range of predicted values and a degree of confidence in the estimate. Many things can be estimated as long as they can be quantified. Examples of this are cost, effort, duration, perhaps level of risk, etc.

Decisions need to be made throughout the life of an initiative from conception to near completion. Therefore, we need estimates that are appropriate to different stages of an initiative.

Estimates are often sought when a solution is conceived to address a set of requirements. A decision is then required on if or how to proceed. Estimates are therefore created based on views of a design. A design can be thought of as a view of a solution expressed as a set of components that are positioned and interrelated in a specific way. The components of a solution are usually delivered as a set of parts (products, services, etc. that are ideally implemented based on proven patterns). If a value can be assigned to each part, then an estimated value for the solution can be created. If parts are associated with, or allocated to, a component, an estimate of the value for a component can be created. The relationships between business, solution, and technology can be considered as seen in figure 1 as one progresses through the spectrum from business (red) through to technology (blue).

In a non-IT example, the business need could be cooked (with requirements making clear what type of cooking is required). The solution is a kitchen, and the parts are the appliances, services, assembly, and integration work required to create the kitchen. In a more traditional IT example, the business need could be to transact business over the Web; the solution, an e-commerce system; and the parts, the various IT components and services necessary to establish and activate the system.

In all cases, in order to calibrate an estimate, a knowledge-base is required that records knowledge of the value of the parts.

Estimation approaches vary over the life cycle

• Over the life cycle of a project, our understanding of the requirements and our design become more detailed. Consequently, our approach to estimation varies.

• In mature domains, building and estimation takes many forms. Someone planning to build a house on an empty lot in a neighborhood could seek cost estimates from:

• A real estate agent—based on what houses in the neighborhood are worth.

• A developer—based on the cost of similar houses.

• A builder—based on costs per square feet and measurements from a simple sketch plan.

• Each building trade—based on the cost of each service (plumbing, wiring, painting, etc.).

• A quantity surveyor—based on a set of measures and a whole table of unit costs (that can be measured from a detailed design).

• A construction company—based on the company’s view of risks (contingency, construction, labor, approval, etc.), funding, etc., plus what the cost would be to construct the dwelling (in light of the company just completing two identical dwellings in the neighborhood).

• The owner—when it has been built based on the actual payments for materials, labor, fees, etc. that were made. (This may or may not reflect all costs, so it could still be considered an estimate.)

It can be seen that a range of estimates, at different points in time, are based on different ways of describing the components and mapping these to knowledge of the costs of parts.

As progress is made through the different levels of design detail, different orthogonal estimation techniques are applied. The estimates are not associated with or assigned to the requirements, e.g., eating, sleeping, bathing, watching TV, etc. in the house.

What We Want in an Estimating Solution

Key decisions that are made based on estimates:

• How can the requirements be altered so the cost (or time, or effort, or risk) can be reduced?

• Can it be demonstrated that the project is worth doing?

• What size team is required to complete the project within an acceptable time frame?

• What contingency is needed to ensure success?

Estimation needs to provide information such as:

• What is the cost/time/effort involved in the initiative?

• What is the estimate, and what is our degree of confidence in it?

• What is the risk associated with the initiative and what impact will this have on an estimate (cost, schedule)?

Some of the key things we want in an estimating solution are to be able to:

• Perform marginal and allocated cost analysis that support decision making.

• Estimate as soon as possible based on what information exists and progressively improve estimates and under variances where they occur.

• Use estimates as a way of assessing risk.

• Quantify the impact on the project of the risk profile.

Traceability and an understanding of allocated cost and marginal cost

A goal of estimation is to understand the cost of a requirement and the cost that would be saved if the requirement were not met. This needs to be possible at various stages throughout a project life cycle. At the start of any project many factors are unknown. And refining the estimates is accomplished by reducing the unknowns.

The many-to-many nature of requirements to components makes it difficult to assign value to an individual requirement, but relatively easy to determine the marginal value of each requirement. A similar situation occurs with components and parts. To enable this, values need to be fully allocated.

Using estimate variances as proxy for risk

A comparison of the results of different estimation techniques will provide an overall measure of risk based on the deviation of the estimates (and, of course, of the range for each estimate). Usually, these variations arise as a result of different potential interpretations of what is required or different assumptions regarding what exists (or is technically feasible). If orthogonal estimation techniques produce a similar estimate, we can infer a high level of understanding and a relatively low level of risk (at this level). If different techniques produce wildly different values, we can conclude the risk is relatively high. One way to improve our confidence is to reduce the discrepancies between orthogonal estimates (and the ability to trace and analyze is often critical to this).

Challenges We Face in Estimating

Estimates are not usually based on requirements

What is often sought is the ability to change the requirements to change the estimate. The subset of the behavior that the solution is to support can be referred to as requirements. In a business context one can consider the behavior of the business as a whole, and the requirements related to specific initiatives of solution.

The way requirements are described consists of the experiential (what we see, use, touch) and the non-experiential (what needs to be there but is not experienced, e.g., safety, sustainability). The existence of non-experiential parameters can make it difficult to intuitively understand the relationships between requirements and components (or parts) in early stages of design. The design process involves elaborating the requirement, i.e., explaining in more detail or more precisely so that the characteristics of components can be better understood. This makes it challenging to base estimates on requirements because many of the factors that influence the design are not explicitly stated as requirements and only emerge when technicians engage in design.

Most estimates are based on a recording of actual values from the past. The records are at different levels of detail (from the cost of a house to the cost of a door) and can be hard to attribute to specific components or parts. Estimates need to be predicated on data that exists (not the data that is wished to exist). As most of the costs recorded are recorded against components or parts (e.g., products, labor, etc.) and not against requirements, often the data doesn’t exist to calibrate estimates based on requirements (unless it is a well-established repeatable process).

In a house, the cost of the kitchen (a component) can be known, but it is hard to determine the cost to support the requirement to “to be able to cook breakfast.” Successful estimation is based on views of what is to be created (components and parts), not the requirements. Attempts to use requirements as key inputs to estimating seldom work well in mature domains with stable technologies and well-defined, straightforward, repeatable integration practices. So it is little surprise that they work poorly in less mature domains with rapidly changing technologies.

Other challenges

In IT other challenges exist:

• IT systems are often assembled from immature technologies.

• IT industry is not well supported by an accepted body of engineering knowledge and best practice. Many organizations struggle to define structural patterns, let alone define the construction processes (and effort, risk, etc.).

• Interfaces between components of an IT system are complex and change as the technologies evolve. SOA is the latest in series of attempts to address this.

• IT industry does not have a habit of retaining knowledge of the time and effort associated with many tasks (or of differentiating between the time taken to learn how to do a task vs. the time taken to do it).

Estimates need to be calibrated based on appropriate facts

Too detailed a decomposition of the components (or activities) can produce bad estimates unless the calibration data exists. A person can tell us how long things take to do if we ask the right level of detail because the person has recorded how long things take at a particular level of detail. Most people know how long it takes to get ready in the morning and then travel to work. But in both cases, if we break the activities into known elemental steps (put on a sock, cross the lobby), we will see that despite the fact that people have done these things thousands of times, they still don’t know how long these individual actions take to do. An estimate on an aggregation of small atomic estimates of poorly calibrated elements is likely to produce a wildly inaccurate result.

New materials, or materials combined in new ways, militate against having sound data to calibrate new estimates on either because no data exists associated with the materials or because we can’t isolate the effort associated with using the materials from the effort associated with learning to use them.

Conclusion and Recommendations

The basis for good estimation is:

• The ability to relate requirements to the components of a solution, where information exists on the values for the parts that will constitute a component.

• The ability to separate the known/understood requirements from the uncertain requirements and use the latter to quantify project uncertainty and risk.

• Maintenance of a catalog of parts (products, pattern, services, etc.) with current recorded values, a key to the calibration of any estimation framework.

• The ability to express and document requirements in a clear and unambiguous manner.

• An understanding of how requirements of an initiative (e.g., project, system) relate to the overall business behavior, so that we can really identify marginal costs.

• Quantification of the risk profile and its impact on the estimates.

To address these issues, it is helpful to create a model and semantically precise estimating framework. Doing so allows:

• Values (cost, effort, time, etc.) to be recorded against defined parts (products, services, or patterns).

• Quantitative risk analysis and an understanding of the confidence of an estimate.

• Business behavior to be related to requirements (so requirements can be validated and gaps and overlaps identified).

• Requirements to be related to components.

• Components to be related to parts (so recorded costs can be used for calibrating estimates).

• All key concepts support decomposition to allow us to understand how orthogonal estimates relate, through different stages of a project.

• Orthogonal views to be dealt and referential and inferential integrity to be analyzed so that inconsistencies between the metrics used in different estimations can be better understood.

If we are to get better at estimating, we also need to look at how we can apply more semantic precision to:

• Requirements discovery and definition.

• Conceptual and logical design.

• Cost tracking and analysis.

by Michael Ellyett, the founder of Enterprise Strategic Transformation and Optimization (www.enterprisesto.com), a consulting firm in the enterprise architecture industry. He can be reach at michael.ellyett@gmail.com