AACE® International Professional Guidance Document No. 02
GUIDE TO QUANTITATIVE RISK ANALYSIS
TCM Framework: 7.6 - Risk Management

Rev. December 20, 2019

 

This document is copyrighted by AACE International and may not be reproduced without permission.

 

Table of Contents

 

Purpose
Introduction
QRA Principles and Method Types – RP 40R-08
QRA Principles and Method Types – Escalation and Currency
Hybrid or Combined Methods and Aggregation of Projects
Complexity, Systems, Behavior, and QRA
Key Terminology – RP 10S-90
Guide to Specific Method RPs (In PGD Map Numerical Order)

1. Understanding the Project System (Stage-Gate Process) and Scope
2. Understanding Estimate Classification (Level of Scope Definition)
3. Value Management: Capitalize on Opportunities
4. Understanding the Base Estimate and Schedules: Basis Documents and Reviews
5. Understanding Concepts and Principles of QRA
6. Quantify the Bias using Validation/Benchmarking
7. Quantify Uncertainty and Risk in Support of Decision Making

Predetermined Guidelines
Simulation (Monte-Carlo Simulation)
Parametric Method
Hybrid Methods
Complexity/Non-Linearity Methods
Escalation
Programs
Management Reserves

8. Application of QRA in Investment Decisions and Change Management
9. Contingency Allocation
10. Contingency Management
11. Maintain Methods and Tools considering Historical Data

Methods Versus Fitness-for-Use Characteristics
References
Contributors

 

 

Purpose

 

Professional guidance documents (PGDs) provide roadmaps to improve use and understanding for collections of related AACE International® (AACE) recommended practices (RPs). They provide clarification on their purpose, function, context, and inter-relationships so that users can locate and focus on those RPs that best meet their requirements. The PGDs also highlight in-progress and preliminarily planned RPs to inform and guide AACE RP development (contact the Decision and Risk Management (DRM) Subcommittee Chair if interested in RP development).

 

PGDs also support the Total Cost Management (TCM) Framework in which RPs relating to each chapter are listed but not described. The heart of most PGDs is a map showing the relationship of the RPs in the context of the subject TCM process (and other TCM processes). This PGD covers RPs for the quantitative risk analysis and contingency (i.e., QRA) steps in the TCM Chapter 7.6 Risk Management process map (re: TCM Figure 7.6-1) [1]. This PGD was developed because of strong interest in the QRA area of practice, particularly in contingency. Many have an established project risk management process, but lack reliable risk quantification as is evidenced in articles on cost overrun and schedule slip research [2,3]. Also, there is a robust set of RPs to choose from in this area, justifying PGD treatment.

 

QRA covers a broad set of practices used primarily to quantify uncertainty and risks in a way that supports investment decision analysis (TCM 3.3), change management decisions (TCM 10.3), and cost and schedule contingency, reserves, escalation and other phased project funding and control basis determinations (TCM 8.1). However, the focus of this PGD is TCM 7.6 which summarizes the QRA step as: “For each risk selected by the team for quantitative risk analysis, a probabilistic estimate of its impact will be prepared. The chosen methodology will have been specified during the risk planning phase.”[1].

 

This PGD also covers fundamental QRA RPs such as for selecting probability density functions as well as highlighting key QRA terms as defined in AACE RP 10S-90 [2]. QRA practices intersect with (and may be part of) base cost estimating and scheduling practices upstream of the QRA (e.g., historical data analysis and estimate and schedule validation to quantify bias) as well as control of the risk funds and contingency durations downstream of the QRA (e.g., contingency drawdown assessment). These related RPs will be highlighted where applicable.

 

It should be noted that PGD-01, Guide to Cost Estimate Classification Systems, addresses the concept of accuracy; a shorthand expression of the uncertainty characteristic of estimates [3]. The classification RPs recommend that accuracy be determined through the application of appropriate QRA methods (i.e., there are no standard accuracy ranges). This guide addresses the appropriateness of the various methods for different project phases and types. Given that accuracy is a key concept and communication challenge, DRAFT RP 104R-19, Understanding Estimate Accuracy [18] may be considered the start of the map and journey into QRA. Next, RP 40R-08, Contingency Estimating – General Principles, covers the principles of recommended contingency and related methods [4]. From there, the QRA RPs cover the alternative methods in detail, and this PGD’s map shows the path to related RPs and the relationship to other TCM processes.

 

Given that a purpose of this PGD is to guide users to RPs that best meet their requirements, the RP descriptions in this document include comparative information. For example, the appropriate method may vary depending on the size of the project, the level of scope development, or the resources available to the organization. No single method addresses all situations. Each RP should address the method’s strengths and weaknesses; this PGD brings some of this information to a single reference. In the end, users must study the RPs themselves to make the best method selection.

 

Where RPs are needed to address the TCM 7.6 process, but not yet developed, they are identified. The intention is to update the PGD regularly as RPs and DRM Subcommittee plans are updated. Where RPs are proposed or in process, key references (primarily AACE Transaction papers and books by RP contributors) are noted in lieu of the pending RP.

 

 

Introduction

 

The AACE Decision and Risk Management Subcommittee began planning an RP series in 2007 with the initiation of a project to develop the Decision and Risk Management Professional (DRMP) certification. An early challenge the team of subject matter experts (SMEs) faced was which of the existing QRA methods to recommend. QRA is an evolving field of practice and not settled science. The approach selected was to first establish the principles that any recommended practice should follow. Then each candidate method was compared to how well it met the principles. Only a small set of methods met the criteria.

 

In all cases, the RPs are aligned with the TCM Framework process for risk management (TCM 7.6) and other TCM processes to which 7.6 is linked. TCM 7.6 is the only industry risk management process that includes explicit steps for QRA. The TCM risk management process map in Figure 1 depicts that when a phase-gate decision is to be made, the RM process loops back through risk assessment for QRA (the step labeled “Analyze Contingency”). On a micro-level, this would be true for minor decisions as well (e.g., decisions made as part of the change management process (per TCM 10.3).

 

Figure 1. Process Map for Risk Management (Figure 7.6-1 from the TCM Framework; 2015 edition)

 

The map in Figure 2 presents the above process in a way that highlights the relationship of the QRA RPs in the context of the related TCM Framework processes. The Figure 2 map highlights key information flow linkages with five main TCM processes or process groups that leverage QRA outputs (note: QRA use is not limited to these):

 

 

Figure 2. QRA RP Relationships and TCM Context

 

The summary QRA process steps in approximate order and respective TCM and RP content to examine are as follows:

 

 

QRA Principles and Method Types – RP 40R-08

 

RP 40R-08, Contingency Estimating – General Principles provides a summary and description of the general principles. In summary, the principles are:

RP 40R-08 describes in detail what is considered with each principle. These are not all repeated here, but as an example of the principle most applicable to the purpose of this PGD, consider the fit-for-use principle description:

 

Fit-For-Use

In addition to considering the general requirements of the client and the process, the practitioner must also consider any other significant contextual characteristics that may or may not affect the estimating practices selected and how they are managed and/or performed. These include, but are not limited to the following:

As should be evident, no single QRA method is best in addressing all these characteristics. This PGD focuses on the above fit-for-use characteristics for each method, while also highlighting how an RP addresses other principles as appropriate.

 

40R-08 also provides a general categorization of common QRA method types, and also has a table of how each general type addresses the principles. This is a starting point for method selection. However, this RP adds to it by comparing specific method RPs against the fitness-for-use principle characteristics. The 40R-08 QRA method types are as follows:

 

QRA Principles and Method Types – Escalation and Currency

 

RP 68R-11, Escalation Estimating Using Indices and Monte Carlo Simulation is currently the only AACE RP for probabilistic escalation risk analysis. This RP outlines the principles for analysis as follows:

RP 68R-11 calls for the use of probabilistic methods for escalation. Unfortunately, this is not yet common practice. Applying a probabilistic method should be considered for projects of long duration (e.g., >18 months) because in volatile economic times, escalation may be the most uncertain cost element on a project and has significant impacts to NPV. The following communication points will help explain to business and finance stakeholders why a probabilistic method may be appropriate:

Currency or exchange rate risk is somewhat related to escalation as both are driven by economic conditions. AACE does not have a currency risk quantification RP. However, the principles and methods used for probabilistic escalation are essentially the same as for currency. Currency adds to the equation the need to forecast future spending not only by cost account, but by currency. Exchange rate indices are then used to assess the uncertainty resulting from varying valuation of currencies as-expended relative to the base estimate currency. Where it is a major risk, finance departments may indemnify the project for currency risk (e.g., via hedging) making it less of an issue.

 

It should be noted that escalation is particularly sensitive to project schedule risk; i.e., schedule slip pushes spending into later years with potentially enormous escalation cost impact. Because schedule slip is quantified as part of contingency QRA, it should be obvious that contingency and escalation quantification are optimally integrated.

 

 

Hybrid or Combined Methods and Aggregation of Projects

 

An implication of following the principles stated is that there is no single method that can be used to quantify all risk on a given project. While a single tool addressing all risk may be possible, such a tool would sacrifice one or more principles. Hence, typical combination of methods will be suggested in this RP.

 

In addition, there are special considerations for and variations of methods when applied in aggregations of projects in a program or portfolio. These topics will also be discussed (note there are no program or portfolio level QRA RPs at this time).

 

As stated, QRA is an evolving area of practice, and combination and integration of methods, and application for aggregations of projects, are areas to pay particular attention to developments. Given that it is evolving, QRA is a great area of practice for volunteers to contribute to RP development.

 

 

Complexity, Systems, Behavior, and QRA

 

Other areas where QRA is evolving include the understanding and quantification of risk relative to:

The trend in industry appears to be towards projects of increasing size or complication (measures of the number of parts) and complexity (measures of the extent and nature of interactions of the parts). There is also greater appreciation of the systems nature of projects for which learnings from systems engineering provide insight. Finally, the impact of human behavior, particularly as evidenced in bias, presents a challenge to quantification.

 

While there are no specific RPs for these topics at this time (e.g., measuring complexity), some QRA methods address these attributes to some extent and this will be noted where applicable (e.g., methods for systemic risks). AACE Transaction papers will be highlighted that focus on these issues (e.g., non-linearity of impacts).

 

 

Key Terminology - RP 10S-90

 

The terminology related to QRA is extensive. One QRA text has over 170 terms in its glossary. AACE RP 10S-90, Cost Engineering Terminology is the recommended reference for QRA terminology. However, there are four terms defining the objects of QRA (the control accounts or activities being determined) worth highlighting in this guide because they are often misunderstood and because there are variations in industry definitions.[1] The AACE definitions evolved from its 60-plus year history and are believed to reflect “what works best” for its members in general cost engineering and project control application. Other industry definitions often contain what the DRM Subcommittee feels are ambiguities (often originating from a qualitative point of view) on one hand, or over-specificity on the other (to suit specific standards that may or may not apply to one’s situation). These issues are discussed. The four object definitions are:


[1] Always refer to the original source document for the latest definition (in this case 10S-90)

 

Allowances

10S-90 Definition: For estimating, resources included in estimates to cover the cost of known but undefined requirements for an individual activity, work item, account or sub‐account.

Discussion:

Allowances are items in the above-the-line base estimate, not contingency. A key phrase is known but undefined which means it is the cost is for a specific (individual) item, not general uncertainties. For example, the requirements for a sump pump’s horsepower may be uncertain (e.g., in range from 10 to 20), but the need for the pump is known. Hence the estimate may include an item titled sump pump allowance with the term allowance flagging the need to better define the item. Allowances may also be specific adders to known items (e.g., concrete waste allowance).

 

General uncertainties (e.g., allowance for weather) should be quantified in contingency. A rule-of-thumb is that if a cost item cannot be physically progressed, it belongs in contingency.

Special Case - Design Development Allowance: This account has evolved into common usage, but the DRM Subcommittee considers including this in base costs to be a poor practice. It is not specific (not progressable). It is believed to have evolved because management has not allowed adequate contingency to address poor scope definition; therefore, estimators allowed for additional contingency above-the-line. If AACE RPs for contingency are used, which include design development risk in contingency, this allowance will result in over-estimation or padding.

 

 

Contingency

10S-90 Definition: An amount added to an estimate to allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and that experience shows will likely result, in aggregate, in additional costs. Typically estimated using statistical analysis or judgment based on past asset or project experience. Contingency usually excludes: 1) Major scope changes such as changes in end product specification, capacities, building sizes, and location of the asset or project; 2) Extraordinary events such as major strikes and natural disasters; 3) Management reserves; and 4) Escalation and currency effects. Some of the items, conditions, or events for which the state, occurrence, and/or effect is uncertain include, but are not limited to, planning and estimating errors and omissions, minor price fluctuations (other than general escalation), design developments and changes within the scope, and variations in market and environmental conditions. Contingency is generally included in most estimates and is expected to be expended.

Discussion: Key words and phrases in this definition include: in aggregate which means only meaningful at the bottom line (not readily amenable to allocation); and expected to be expended which literally means the mean of the QRA outcome distribution; and changes within the scope which puts an emphasis on the team to define what scope means (scope will differ between the owner and contractor). It states design developments are part of contingency. It also clearly delineates contingency from management reserves and escalation.

Special Case - Earned Value Management Based Upon ANSI EIA 748: An amount held outside the performance measurement baseline (PMB) for owner level cost reserve for the management of project uncertainties. This definition creates ambiguity (is it contingency or reserve?; is it expected to be expended?) and applies only when standardized EVM is applied (but it must be understood if one is working in that environment. See RP 75R-13 for EVMS considerations.)

 

 

Management Reserve

10S-90 Definition: An amount added to an estimate to allow for discretionary management purposes outside of the defined scope of the project, as otherwise estimated. May include amounts that are within the defined scope, but for which management does not want to fund as contingency or that cannot be effectively managed using contingency.

Discussion: Key words and phrases in this definition include: Management which means authority lies above the project manager level. Also discretionary and outside the defined scope allows wide latitude to that management. The phrase cannot be effectively managed using contingency explicitly addresses the common situation of low-probability, high-impact risk events for which partial funding (e.g., p50 or mean) is illogical. It also clearly delineates management reserves from contingency and escalation.

Special Case - Earned Value Management Based Upon ANSI EIA 748: An amount held outside the performance measurement baseline (PMB) to handle unknown contingency at the total program level. Management reserve has no scope, is not identified to specific risks, and is not time‐phased. It is typically not estimated or negotiated and is created in the budget development process. This definition is unnecessarily limiting by restricting management discretion to fund (or not) specific risks not amenable to contingency management. It applies only when standardized EVM is applied (but it must be understood if one is working in that environment. See RP 75R-13 for EVMS considerations.)

 

 

 

Escalation

10S-90 Definition: A provision in costs or prices for uncertain changes in technical, economic, and market conditions over time. Inflation (or deflation) is a component of escalation.

Discussion: Key words and phrases in this definition include over time which differentiates economic driven trends from discrete risk events; e.g., gradually improving but more costly technology is escalation while an instantaneous step change in technology and price would be a risk event and in contingency or management reserves. The term conditions implies pervasive, over-arching background such as would affect all projects of a type or in a region. It also differentiates escalation from inflation.

 

There are also several terms defining risk types being quantified that are sometimes misunderstood. The AACE definitions evolved specifically from the DRM Subcommittee’s focus on fit-for-use QRA methods. Again, an emphasis has been on avoiding ambiguity originating from qualitative points of view. There are two term-pairs discussed:

 

Systemic versus Project-Specific Risks

10S-90 Definitions:

Systemic Risk: A risk taxonomy designation used to classify project risks for the purposes of selecting a quantification method (i.e., contingency determination). Systemic risks are uncertainties (threats or opportunities) that are an artifact of an industry, company or project system, culture, strategy, complexity, technology, or similar over‐arching characteristics.

 

Project-Specific Risk: A risk taxonomy designation used to classify project risks for the purposes of selecting a quantification method (i.e., contingency determination). Project‐specific risks are uncertainties (threats or opportunities) related to events, actions, and other conditions that are specific to the scope of a project. (e.g., weather, soil conditions, etc.). The impacts of project‐specific risks are more or less unique to a project.

Discussion: These terms are for purposes of selecting specific QRA methods. The implied linking of QRA to systems thinking is intentional; the impact of system attributes (systemic risks) by nature is best studied and quantified through empirical research of project systems for which there is considerable research available. Systemic risks are uncertainties (probability = 100%) with impacts usually derived empirically. Project-specific risks may be uncertainties (probability = 100%) or risk events (probability < 100%) which can be quantified effectively with simulation methods; i.e., they are amenable to project team judgment and experience as to probability and impact.

 

Uncertainties versus Risks Events

In risk management, there is a host of risk taxonomy terms in use for many purposes and whose meanings are directed more towards qualitative than quantitative analysis. For QRA, for methods using probability x impact in some manner, the most meaningful definitions for risk and uncertainty are (note: these are not in RP 10S-90 and readers should monitor terminology development):

 

Uncertainty: A risk with probability of occurrence of 100%; i.e., a project condition, fact or attribute. It does not occur, but is a state of being which results in uncertain outcomes. Inherent risk or issues are sometimes used as synonyms.

 

Risk Event: A risk with probability of occurrence less than 100%. Contingent risk is sometimes used as a synonym.

Discussion: This taxonomy is useful but does not in itself drive QRA method selection; e.g., one can use the expected value method (probability x impact) for either uncertainties or risk events, only with different probability values. As an example of how these terms affect QRA practice, some references suggest not quantifying issues when in fact they may be the main driver of uncertain outcomes.

 

 

Guide to Specific Method RPS (IN PGD MAP NUMERICAL ORDER)

 

The following table summarizes the various method RPs available for various QRA purposes. The summary QRA process steps outlined in the Introduction and Figure 1 is used to organize this listing as follows. Where there are no RPs, AACE Transaction papers or other sources are suggested. In the section that follows this, a table is provided that compares the primary QRA methods for contingency in respect to fitness-for-use characteristics.

 

1. Understanding the Project System (Stage-Gate Process) and Scope

TCM 4.1 - Project Implementation

No RPs

Discussion: An owner’s capital project system (including integration with its suppliers) and its effectiveness (or lack thereof) is a major source of uncertainty in cost and schedule outcomes. Understanding the system through which a project is being performed, and its strengths and weaknesses (systemic risk) is essential grounding for any QRA analysis. An essential element of that is the stage or phase-gate process which guides business framing and project scope definition, including at the portfolio level if applicable.

Selected References: [5, 6]

  • Barshop, Paul, “Capital Projects: What Every Executive Needs to Know to Avoid Costly Mistakes and Make Major Investments Pay Off”, John Wiley & Sons, Inc., Hoboken, New Jersey, 2016.
    PURCHASE ON AMAZON

  • Stephenson, L. and M. Pruneau, “Strategic Portfolio Management: Improving Capital Utilization and Competitive Advantage”, 2016 AACE International Transactions, TCM-2110, AACE International, Morgantown WV, 2016.
    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

 

 

2. Understanding Estimate Classification (Level of Scope Definition)

TCM 7.3 - Cost Estimating and Budgeting
Professional Guidance Document (PGD) No. 01: Guide to Estimate Classification Systems

Discussion: PGD-01, provides a roadmap to the various classification RPs. Classification defines the scope definition deliverables required as the basis for cost estimates of each class. The classes align with prevailing industry stage-gate system phases. Research shows that the level of scope definition, an artifact of the project system (systemic risks), is a major source of uncertainty in cost and schedule outcomes and driver of accuracy ranges. Hence a scope definition rating such as classification is a necessary input for QRA analysis.

 

 

3. Value Management: Capitalize on Opportunities

TCM 11.5 - Value Management and Value Improving Practices (VIPs)
RP 30R-03: Implementing Project Constructability
RP 48R-06: Schedule Constructability Review

Discussion: There are many practices that have a specific focus and/or significant effect on getting the most value from the project scope and plans. Some call these value improving practices or VIPs; they are applied as part of stage-gate systems. In risk management terms, VIPs are focused on capitalizing on opportunities. If risk identification finds a potential opportunity, it should be referred to a value function to apply VIPs as appropriate. Each VIP, be it value engineering/value analysis (TCM 7.5), constructability (RPs 30R-03 and 48R-06) or others, includes making decisions as to recommended changes or improvements to the scope (changes may be channeled through the change management process if in effect). Each change resulting from a VIP also carries risk, which needs to be considered using QRA. Constructability is the most common VIP used in engineering, procurement and construction.

Selected References: [7]

  • Lozon, Jim and G. Jergeas, “The Use and Impact of Value Improving Practices and Best Practices”, Cost Engineering, Vol. 50, No. 06, AACE International, Morgantown, WV, 2008.
    MEMBER DOWNLOAD | JOIN AACE

 

 

4. Understanding the Base Estimate and Schedules: Basis Documents and Reviews

TCM 7.3 - Cost Estimating and Budgeting
TCM 7.2 - Schedule Planning and Development
RP 31R-03: Reviewing, Validating, and Documenting the Estimate
RP 78R-13: Original Baseline Schedule Review - As Applied in Engineering, Procurement, and Construction
RP 34R-05: Basis of Estimate
RP 38R-06: Documenting the Schedule Basis

Discussion: QRA starts with understanding the base estimate and schedule. To the extent the base is biased (i.e., aggressive or conservative) and to the extent it includes allowances for uncertainty and risk, explicit or not, the need for contingency and other risk funds and duration will vary. A well done basis document will describe the planning strategy, assumptions, allowances, and other important information. Estimate and schedule reviews will further surface quality issues (systemic risks). Validation and benchmarking (covered later) will surface and quantify bias issues.

 

 

5. Understanding Concepts and Principles of QRA

TCM 7.6 - Risk Management
RP 40R-08: Contingency Estimating - General Principles
RP 66R-11: Selecting Probability Distribution Functions for use in Cost and Schedule Risk Simulation Models
RP (DRAFT) 104R-18:
Understanding Estimate Accuracy
RP 75R-13: Schedule and Cost Reserves within the Framework of ANSI/EIA-748
RP -TBD-: Complexity Assessment

Discussion: These RPs cover elements of QRA; their titles are self-explanatory. They are not QRA methods in themselves, but cover general principles and inputs (e.g., PDFs) and outputs (e.g., accuracy) that apply to all QRA methods to some extent.

 

 

6. Quantify the Bias using Validation/Benchmarking

TCM 6.1 - Asset Performance Assessment
TCM 7.2 - Schedule Planning and Development
TCM 7.3 - Cost Estimating and Budgeting
RP -TBD-: Estimate and Schedule Validation and Benchmarking

Discussion: Every estimate and schedule is biased in some way. Establishing a cost and schedule strategy is a way to plan bias to meet strategic objectives (see: Basis). Whether planned or not, it is necessary to quantify the bias as the first quantitative step of QRA because aggressiveness (optimism) or conservativeness (pessimism) in the base adds or deducts from the risk funds and durations to some extent. Also, aggressiveness can be a risk driver in its own right. Validation is a quantitative practice for measuring bias (and to some extent quality) of the estimate and schedule. Validation relies mainly on comparisons of project cost and schedule metrics to historical metrics. Benchmarking is similar, but focuses on comparison to metrics for external projects or project systems with a focus on competitiveness of the business.

 

 

7. Quantify Uncertainty and Risk in Support of Decision Making

TCM 7.6 - Risk Management

Discussion: Each method is discussed in its own sub-section by general type of method. Also see the fitness-for-use comparison table in later in this document.

 

Predetermined Guidelines

No RP

Discussion: This type of method may vary from simple contingency percentages (e.g., 20, 15, and 10 percent for Class 5, 4 and 3) to elaborate multi-dimensioned tables of percentages that vary by other parameters such as the level of project technology. There are no RPs because this method is a tabular version of the parametric (RP 42R-08) method for systemic risks. Unfortunately, pre-determined percentages are often not realistic and have little empirical basis; such values are never recommended. If such tables are to be used, the DRM Subcommittee recommends that the parametric method be used (for which reliable models are available from AACE in RP 43R-08) to generate such values and tables. As such, this is not a separate method; just a way of reporting and applying the parametric results.

 

Simulation (Monte-Carlo Simulation)

General Discussion: Monte-Carlo simulation (MCS) is an inferential statistical method that generates a distribution of possible outcomes from a mathematical model. It does so by sampling from model variables for which fixed values have been replaced with distributions of possible values (see RP 66R-11 for distribution types) while considering dependencies between the model variables where they are not independent. There are many possible model variations, but the two models that address the RP 40R-08 principles, and are covered by RPs, are a risk-driven, cost-loaded critical path method (CPM) network model where a quality CPM network model is available (57R-09), and the risk-driven expected value method where a quality CPM in not available or practical (65R-11). Both integrate the analysis of cost and schedule; a key principle of RP 40R-08. Being integrated, both support joint confidence level (JCL) reporting (note: JCL is a way of reporting and not a method in its own right) [8]. See the discussion on hybrid method application in respect to optional ways that systemic risks may be evaluated using parametric methods and then integrated into either a CPM or expected value simulation.

Note on Methods Not Covered by RPs: There are several MCS based methods in common use that are not covered by RPs because they fail to adequately address key RP 40R-08 principles. They are:

 

Non-Risk Driven Ranging: A non-risk driven ranging model is the cost estimate itself (in detail or summary levels) or the CPM schedule (in detail or summary levels) with the MCS approach being to replace fixed cost or duration values with 3-point distributions determined from team brainstorming or interview input. They do not explicitly link the risks to the variable impacts; a key 40R-08 principle. Being implicit and subjective, research has shown the results are not realistic when there are significant risks and reliable QRA is needed most [9].

 

CPM Schedule to Cost Ranging: Sometimes called schedule risk analysis to cost risk analysis (SRA2CRA). This method analyzes schedule risk in one analysis, followed by cost risk in a separate later analysis. The separation of analyses in time tends to diminish integration and/or limits consideration of cost-schedule trading. No benefit is gained from splitting the analysis process in time, effort, or output that justifies possibly sacrificing the fidelity of integrated risk-driven, cost-loaded CPM or integrated expected value.

RP 57R-09: Integrated Cost and Schedule Risk Analysis Using Risk Drivers and Monte Carlo Simulation of a CPM Model

Plus: RP 64R-11: CPM Schedule Risk Modeling and Analysis: Special Considerations (Adjunct)

57R-09 Discussion: Per the RP introduction: The methods are based on integrating the cost estimate with the project schedule by resource-loading and costing the schedule’s activities. The probability and impact of risks/uncertainties are specified and the risks/uncertainties are linked to the activities and costs that they affect. The method makes optimal use of the schedule model; the analysis of how a schedule behaves under risk results in an improved, risk-robust plan (i.e., the practice could be considered a risk-aware schedule development practice as much or more so than QRA).

64R-11 Discussion: This is an adjunct to 57R-09; not a stand-alone method. Per the text: This RP discusses key procedural, analytical and interpretive considerations in preparation and application of a CPM model; considerations that were not covered in the broader methodological RPs (i.e., 57R-09). Topics include schedule quality, schedule model building, linking risks to activities, and interpreting results. The rigor and complexities of CPM based methods may lead one to use the less imposing EV method (65R-11) for less strategic projects.

RP 44R-08: Risk Analysis and Contingency Determination Using Expected Value (EV)

44R-08 Discussion: The method in this RP addresses cost risk only; it is offered as a step improvement from cost ranging which fails to link risks to impacts. With EV, the probability of occurrence of each critical risk is multiplied by a cost impact distribution (p x i) to calculate an expected value. The sum of all risk EV results is calculated and MCS is applied to the model. The RP covers various analytical tasks and challenges. RP 65R-11 extends this method to provide integrated cost and schedule risk analysis using EV and is recommended over using entry level 44R-08 alone. Note: this RP may be deleted and its content integrated into 65R-11

RP 65R-11: Integrated Cost and Schedule Risk Analysis and Contingency Determination Using Expected Value (EV)

65R-11 Discussion: This RP extends the EV method in 44R-08 to add analysis of the schedule duration impact of identified critical risks. The cost and schedule impact of each risk is based on a conceptually planned risk response (what will be done if the risk occurs). As such it is an integrated cost and schedule method (i.e., it can produce a joint confidence level), albeit without direct use of a CPM model. It is a practical method for non-strategic projects and/or those without a quality CPM (schedule analysis is intuitive).

 

Parametric Method

RP 42R-08: Risk Analysis and Contingency Determination Using Parametric Estimating

RP 43R-08: Example Models as Applied for the Process Industries

Discussion:

Per the RP text: ...the method is used to estimate contingency based on risk parameters (e.g. level of scope definition, process complexity, etc.). This RP includes practices for developing the parametric methods and models (generally empirically-based). The method covers parametric modeling of both cost and schedule outcomes, but it is focused on quantifying systemic risks which have fairly universal implications. Being regression-based, the method is probabilistic, but unlike with MCS the data is actual rather than simulation generated. The parameters in a model are quantitative ratings of systemic risks (e.g., classification number is an example rating). The method can be used alone at early phases, but is typically applied in combination with a simulation method to address the systemic risks (see: Hybrid Methods). The method produces single distributions for cost and schedule; the challenges this presents to contingency allocation and management are discussed in those sections. It is expected that this RP will evolve as machine learning and artificial intelligence replace regression-based analysis of data.

 

RP 43R-08 is an adjunct to 42R-08 and includes working Excel® versions of the John Hackney and Rand Corporation parametric models. The Hackney model covers cost, while Rand provided models for both cost and construction schedule.

 

Hybrid Methods

No RPs

Discussion: There are two approaches in use to quantify systemic risks (where such risks are seen as worthy of special attention). There are differing opinions on the applicability of empirically-based models.

 

Subject Matter Expert (SME) Assessment: Identify, rate and quantify systemic risks like any other in a simulation; however, with special concern for their sensitive nature (e.g., judging the competency of the team) and challenges with quantifying impacts that can be insidious in nature (e.g., interacting with other risks and effecting any or all items and activities).

 

Parametric Method: See Parametric Method section. In this case, the systemic risks are segregated from project-specific risks and uncertainties, rated and quantified using a parametric model. The probabilistic outputs of the parametric model for cost and schedule are then carried over to a simulation as the first specific risk (as an uncertainty, probability = 100%). This approach is called a hybrid method. As there are two basic, integrated simulation methods: CPM and EV, there are two basic hybrid approaches:

It is anticipated that RPs will be developed for these two hybrid methods, and the treatment of systemic risks will likely be added as an addenda to 57R-09. The references below are for papers designed as potential references to these RPs.

References: [10, 11, 12]

  • SME Assessment: Hulett, David and W. Whitehead, “The Monte Carlo Method for Modeling and Mitigating Systemic Risk”, 2016 AACE International Transactions, RISK-2142, AACE International, Morgantown WV, 2016.
    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

  • CPM Based Hybrid: Cropley, Colin, “Modelling Realistic Outcomes Using Integrated Cost and Schedule Risk Analysis”, 2017 AACE International Transactions, RISK-2510, AACE International, Morgantown WV, 2017.
    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

  • EV Based Hybrid: Hollmann, J, “Realistic and Practical Project Risk Quantification (without CPM)”, 2107 AACE International Transactions, RISK-2515, AACE International, Morgantown WV, 2017.
    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

 

Complexity/Non-Linearity Methods

No RPs

Discussion: Empirical research has indicated that cost and schedule outcome distributions of actual/estimate metrics are sometimes not well modeled by traditional QRA simulation (e.g., actual distributions sometimes exhibit bimodality, fat/long overrun tails, etc.). It is hypothesized that as complexity and other systemic risks increase, the failure mode of projects can shift from orderly to disorderly or chaotic (e.g., blowouts). Methods have been published that apply methods of measuring or rating complexity, and interaction with other risks, and applying that to QRA models in a way that generates non-linear outcomes such as seen in some empirical research. There are two methods of addressing systemic risks as covered elsewhere (including complexity): SME assessment and hybrid. The references below cover background research on the topic as well as potential complexity/non-linear QRA RPs as our knowledge increases.

Selected References: [13, 14, 15]

  • Research: Ogilvie, Alexander (IPA, Inc.), “A Tale of Two Tails: Chaos in Estimating Predictability”, 2016 AACE International Transactions, EST-2099, AACE International, Morgantown WV, 2016.
    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

  • SME Assessment: Raydugin, Yuri, “Non-linear Probabilistic (Monte-Carlo) Modeling of Systemic Risks”, 2018 AACE International Transactions, RISK-2808, AACE International, Morgantown WV, 2018.
    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

  • Hybrid: Hollmann, John, “Project Risk Quantification", Chapter 14: The Tipping Point: Risk Analysis at the Edge of Chaos”, Probabilistic Publishing, Sugar Land TX, 2016.
    PURCHASE ON AMAZON

 

Escalation

58R-10: Escalation Principles and Methods Using Indices
68R-11: Escalation Estimating Using Indices and Monte Carlo Simulation

Discussion:

From the RP text: Defines basic principles and methodological building blocks for estimating escalation using forecasted price or cost indices while also addressing uncertainty using Monte Carlo simulation. RP 58R-10 describes a deterministic or base estimating method while RP 68R-11 is an addendum adding probabilistic and scenario/sensitivity analysis to the base method. The RP starts with extending the RP 40R-08 principles to cover escalation.

 

A feature of 68R-11 is that it uses the distribution outputs from cost and schedule contingency models as inputs to the probabilistic escalation model. This integration produces outcome distributions of not only escalation cost, but total capex including cost and schedule contingency; this universal capex distribution can then be readily applied in probabilistic NPV models.

 

Note that currency risks can be estimated in a similar method as for escalation; an RP addendum has been suggested.

 

Programs

No RPs

Discussion: As stated in TCM 7.1: Sometimes a project is part of a program or set of related projects. A program can be viewed as the top level on the WBS with the next level including the projects within the program. The basic project control concepts apply to a program with the added complication of integrating its projects. This same complication arises with contingency; each project manager in a program requires a contingency budget, and so to should the program director to address integrative risks that manifest themselves at the program level. This could be in addition to management reserve. As with control, the same QRA concepts apply to a program as for projects, just with different (or compounded) risks. There is no RP at this time for program level contingency determination. The reference below proposes a method and potential RP using the EV hybrid method.

Selected References: [15]

  • Programs Using EV Hybrid: Hollmann, John, “Project Risk Quantification", Chapter 7: Introduction to Risk Quantification Methods, Probabilistic Publishing, Sugar Land TX, 2016.
    PURCHASE ON AMAZON

 

Management Reserves

No RPs

Discussion: The definition of management reserves (MR), and its distinction from contingency, was discussed earlier. However, specific methods for MR quantification are not covered by RPs. For MR that is purely discretionary, so to are the methods. However, for low-probability, high impact risks not amenable to contingency funding (e.g., p x i is illogical) is covered by the following reference. In general, it suggests removing those risks from the contingency QRA and probabilistically analyzing each one separately for decision making by the business.

Selected References: [15, 16]

  • Using EV Hybrid Model: Hollmann, John, “Project Risk Quantification", Chapter 12: Project Specific Risks and the Expected Value Method, Probabilistic Publishing, Sugar Land TX, 2016.
    PURCHASE ON AMAZON

  • Using CPM-Based Model: Vinueza C, Gustavo, “Optimizing Your Management Reserve”, AACE International Transactions, AACE International, Morgantown WV, 2019.

    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

 

 

8. Application of QRA in Investment Decisions and Change Management

TCM 3.3 - Investment Decision Making
TCM 10.3 - Change Management

No RPs

TCM 3.3 - Investment Decision Making: Decision analysis of alternatives in consideration of risk as covered by TCM 3.3 is a separate topic. Decision analysis involves more inputs than the capital cost and schedule risks covered by the QRA methods in this RP (e.g., must consider revenue and opex as well as capex in NPV analysis). However, there are three areas where alignment between business decision analysis and project risk analysis is worth highlighting. One is that the start of the revenue stream in NPV analysis is directly driven by the project completion date quantified in project QRA. Another is that capex price escalation and revenue (sell price) are driven by economic conditions and hence their analysis should be based on common assumptions and scenarios. Finally, the nature of a business’s interaction with stakeholders and project teams and the efficacy of its capital management system are systemic risks to a project; i.e., the business owns most project systemic risks to some extent. As was noted, the probabilistic escalation method generates a universal capex cost distribution that can be readily used in NPV models.

TCM 10.3 - Change Management: The project change management process covered in TCM 10.3 includes a step for evaluating the risks of trends, deviations and proposed changes to support change management decisions. While scope changes may be elevated to business investment decision making, most change management decisions are tactical and limited in nature and involve less rigorous analysis. The analysis may include elements of formal QRA, but there is no RP for QRA during execution. However, change management processes often call for full QRA to be conducted as part of periodic, milestone-based, or as needed assessment during execution. See Contingency Management for QRA practices during execution.

 

 

9. Contingency Allocation

TCM 8.1 - Project Control Plan Implementation

General Discussion:

The RP 10S-90 definition of contingency expresses or implies two principles relative to the issue of allocation. The first principle is that contingency is only meaningful in aggregate (aggregation principle). The second principle is that contingency is an account under the spending authority of the project manager or high level WBS element manager (authority or control principle).

 

The aggregation principle derives from the fact that risk quantification and contingency determination are stochastic and hence cannot be reliably allocated to lower level control accounts (e.g., activities, items or disciplines). While it is technically possible to drive project-specific risk impact quantification calculations to increasingly finite levels of cost or activity impact detail, this increasingly implies deterministic behavior. Quantifying systemic risk impacts is even more stochastic and less reliable.

 

The authority principle means high-level WBS managers with wide authority may require contingency to deal with risks as they occur in the sub-project in their control. This justifies allocating contingency at an appropriate aggregation level. For this reason, the only RP for this topic is about contract risk allocation, not low-level cost account allocation which is generally not recommended.

 

For analysis by and allocation to sub-projects, QRA methods applicable to programs apply.

RP 67R-11 Contract Risk Allocation as Applied in Engineering, Procurement, and Construction

Discussion: As stated in RP 67R-11: The burden of contingency funding and its management may fall to different entities. Whoever underwrites the risk of cost or schedule overrun must be capable of utilizing the appropriate contingency and, irrespective of the contractual arrangement, the principles of contingency derivation and management remain unchanged. This RP covers the principles of allocating risks, and associated funds and durations, to contractors; primarily a risk treatment practice. QRA practices must consider the nature and manner of this allocation.

 

 

10. Contingency Management

TCM 8.1 - Project Control Plan Implementation
TCM 10.3 - Change Management

No RPs

Discussion: Once a budget for contingency has been determined, there are quantitative steps for its time-phased budgeting and then for later contingency update. During execution, the use of contingency, as controlled by the change management process working with risk management, is plotted against a contingency time-phased spending curve (draw-down curve) for reporting and assessment. There are two general approaches to contingency management that reflect differences in viewpoint as to the nature of risk behavior and its manageability. The words stochastic and deterministic are used to distinguish them although both approaches employ probabilistic methods, and continually manage risks (e.g., maintain risk registers, etc.) during execution.

 

Stochastic methods assume the primacy of the aggregation principle (see Contingency Allocation); i.e., that QRA methods are highly uncertain approximations (e.g., a Class 5 estimate) that are not amenable to allocation either by account or risk. As such, just as contingency cannot be allocated by account, its consumption cannot be rigorously forecast against time. A common stochastic method is to plot drawdown in proportion to spending of the base accounts known to be particularly uncertain (e.g., proportional to labor and labor-related spending). During execution, if the identified risks change materially and/or contingency use is out-of-line with the plan curve, and at a minimum at pre-determined milestones, overall QRA is re-done for the scope of work remaining (including systemic risks which may degrade during execution). The approach is not dependent on the quality of the base estimates and schedules or rigor of control practices.

 

Deterministic methods typically assume that QRA calculations establish a meaningful basis for planning contingency consumption, risk-by-risk. That includes assuming systemic risks and their impacts can be parsed (i.e., it is not amenable to the parametric QRA method). Further, those using this method also tend to track contingency drawdown by using the risk register as a mechanistic control tool whereby when a risk is no longer active, it’s cost impact estimate is subtracted from remaining contingency. As risks evolve and as new risks are added, the plan is revised as determined by risk-by-risk quantification. Overall QRA may be re-assessed periodically or as needed. The approach is dependent on high quality base estimates and schedules and rigorous control practices.

Selected References: [15, 17]

  • [Stochastic] Hollmann, John, “Project Risk Quantification", Chapter 17: Budgeting for Risks and Account Control , Probabilistic Publishing, Sugar Land TX, 2016.
    PURCHASE ON AMAZON

  • [Deterministic] Stroemich, Christopher, and M. Dissanayake, “Project Risk Drawdown: A Structured Approach to Contingency Management”, AACE International Transactions, AACE International, Morgantown WV, 2018.
    MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

 

 

11. Maintain Methods and Tools considering Historical Data

TCM 10.4 - Project Historical Database Management

No RPs

Discussion: The use of historical data is a principle of QRA methods per RP 40R-08. This is particularly true for parametric modeling which is directly based on historical data. However, other methods rely on data to some extent. For example, history should inform the factors in predetermined guidelines, provide lessons learned and typical risk behavior for simulation analysis. Also, weather history is often used to inform quantification of weather event risks. It is also essential for the development of comparison metrics for validation methods. The capture and management of data to support QRA is the same process as for base estimating and schedule development purpose. An area of current and future practice development in respect to data analysis is machine learning and AI. Pending development of an RP covering databses and risk needs, the following reference is provided.

References: [15]

  • Hollmann, John, “Project Risk Quantification", Chapter 18: Closing the Loop, Probabilistic Publishing, Sugar Land TX, 2016.
    PURCHASE ON AMAZON

 

 

Methods Versus Fitness-for-Use Characteristics 

 

The following table provides a summary comparison of the base QRA methods in comparison to the RP 40R-08 fitness-for-use characteristics. Note that hybrid method RPs integrating systemic risk quantification into the CPM and EV methods are in development.

 

Characteristic

57R-09 (CPM)

65R-11 (EV)

42R-08 (Parametric)

Project scope (size, complexity, technology).

Strategic projects and programs.

Moderate to large projects and programs.

Any project or program including small.

Risk type: systemic vs. project-specific.

Project-specific and systemic (hybrid).

Project-specific and systemic (hybrid).

Systemic only.

Project phase: estimate/schedule class.

Class 4 onward.

Class 4 onward.

All classes, including Class 5.

Base estimate/schedule methods.

High quality, resource-loaded CPM required.

Amenable to any base estimate/schedule quality.

Amenable to any base estimate/schedule quality (quality is a risk driver).

Special skills and knowledge of risk SME (assumes basic facilitation skills and project knowledge).

Significant scheduling expertise. Systemic interview elicitation skill if applicable.

None, systemic interview elicitation skill if applicable.

None for application, but statistical analysis for model calibration.

Table 2: Primary QRA Methods Versus Fitness-for-Use Characteristics

 

 

References

 

[1]

Stephenson, L. (editor), AACE International Total Cost Management Framework, AACE International, Morgantown, WV (latest revision)
MEMBER DOWNLOAD | NON-MEMBER PURCHASE | JOIN AACE

[2]

AACE International, Recommended Practice 10S-90, “Cost Engineering Terminology”, AACE International, Morgantown, WV (latest revision)
MEMBER DOWNLOAD | NON-MEMBER DOWNLOAD | JOIN AACE

[3]

AACE International, Professional Guidance Document (PGD) No. 01, “Guide to Estimate Classification Systems”, AACE International, Morgantown, WV (latest revision)
MEMBER DOWNLOAD | NON-MEMBER DOWNLOAD | JOIN AACE

[4]

AACE International, Recommended Practice 40R-08, “Contingency Estimating – General Principles”, AACE International, Morgantown, WV (latest revision)
MEMBER DOWNLOAD | NON-MEMBER PURCHASE | JOIN AACE

[5]

Barshop, Paul, “Capital Projects: What Every Executive Needs to Know to Avoid Costly Mistakes and Make Major Investments Pay Off”, John Wiley & Sons, Inc., Hoboken, New Jersey, 2016.
PURCHASE ON AMAZON

[6]

Stephenson, L. and M. Pruneau, “Strategic Portfolio Management: Improving Capital Utilization and Competitive Advantage”, 2016 AACE International Transactions, TCM-2110, AACE International, Morgantown, WV, 2016.
MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[7]

Lozon, Jim and G. Jergeas, “The Use and Impact of Value Improving Practices and Best Practices”, Cost Engineering, Vol. 50, No. 06, AACE International, Morgantown, WV, 2008.
MEMBER DOWNLOAD | JOIN AACE

[8]

Steiman, Samuel and Dr. David T. Hulett, “Identifying the Most Probable Cost – Schedule Values from a Joint Confidence Level (JCL) Risk Analysis”, AACE International Transactions, RISK-3111, AACE International, Morgantown, WV, 2019.

MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[9]

Burroughs, Scott and G. Juntima, “Exploring Techniques for Contingency Setting”, 2004 AACE International Transactions, EST.03, AACE International, Morgantown, WV, 2004.
MEMBER DOWNLOAD | JOIN AACE

[10]

Hulett, David and W. Whitehead, “The Monte Carlo Method for Modeling & Mitigating Systemic Risk”, 2016 AACE International Transactions, RISK-2142, AACE International, Morgantown, WV, 2016.
MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[11]

Cropley, Colin, “Combining Parametric and CPM-based Integrated Cost-Schedule Risk Analysis”, AACE International Transactions, RISK-3037, AACE International, Morgantown WV, 2019.

MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[12]

Hollmann, J, “Realistic and Practical Project Risk Quantification (without CPM)”, 2017 AACE International Transactions, RISK-2515, AACE International, Morgantown, WV, 2017.
MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[13]

Ogilvie, Alexander (IPA, Inc.), “A Tale of Two Tails: Chaos in Estimating Predictability”, 2016 AACE International Transactions, EST-2099, AACE International, Morgantown, WV, 2016.
MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[14]

Raydugin, Yuri, “Non-Linear Probabilistic (Monte Carlo) Modeling of Systemic Risks”, 2018 AACE International Transactions, RISK-2808, AACE International, Morgantown, WV, 2018.
MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[15]

Hollmann, John, “Project Risk Quantification”, Probabilistic Publishing, Sugar Land TX, 2016.
PURCHASE ON AMAZON

[16]

Vinueza C, Gustavo, “Optimizing Your Management Reserve”, AACE International Transactions, OWN-3236, AACE International, Morgantown WV, 2019.

MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[17]

Stroemich, Christopher, and M. Dissanayake, “Project Risk Drawdown A Structured Approach to Contingency Management”, 2018 AACE International Transactions, RISK-2795, AACE International, Morgantown, WV, 2018.
MEMBER DOWNLOAD | WATCH VIDEO (MEMBER) | JOIN AACE

[18] AACE International, Recommended Practice (DRAFT) 104R-18, “Understanding Estimate Accuracy”, AACE International, Morgantown, WV (latest revision)
MEMBER DOWNLOAD | NON-MEMBER PURCHASE | JOIN AACE | WATCH VIDEO

 

  

Contributors

 

Disclaimer: The content provided by the contributors to this professional guidance document is their own and does not necessarily reflect that of their employers, unless otherwise stated.

 

John K. Hollmann, PE CCP CEP DRMP FAACE Hon. Life (Primary Contributor)
Dr. Manjula Dissanayake, CCP

 

This document is copyrighted by AACE International and may not be reproduced without permission.