GAO Earned Value Management (EVM) Audit Findings Based on Best Practices for EVM in the GAO Cost Estimating and Assessment Guide Karen Richey December 2012
EVM is an Important Management Decision Support Tool EVM indicates how past performance may affect future performance The data isolates cost and schedule variances by WBS elements allowing for: An understanding of technical problems Opportunities to reallocate effort to mitigate risk or address issues The two main purposes for implementing an EVM system are to: 1. Encourage the use of effective internal cost and schedule management controls 2. Allow the customer to rely on timely and accurate data for determining contract performance 2
The Thirteen Steps in the EVM Process 1. Define the scope of work using a WBS 2. Identify who in the organization will perform the work 3. Schedule the work 4. Estimate the labor and material required and authorize budgets including MR 5. Determine objective measure of earned value 6. Develop the performance measurement baseline 7. Execute the work plan and record all costs 8. Analyze EVM performance data and record variances from PMB plan 9. Forecast EACs using EVM 10. Conduct an integrated cost-schedule risk analysis 11. Compare EACs from EVM in Step 9 with EAC from risk analysis in Step 10 12. Take management action to mitigate risks 13. Update the PMB as changes occur 3
EVM Data Should be Examined for Reliability Before Using It to Make Decisions or Calculate EAC Projections For EVM data to be of any value it must be reliable The data should be generated by a system that has been deemed compliant with the ANSI 32 guidelines The performance measurement baseline should be validated by an Integrated Baseline Review in a timely manner EVM surveillance by independent and qualified staff should be continually monitoring the implementation of the system The contractor s financial accounting system has received an unqualified opinion Data anomalies like negative values for BCWS, BCWP, and ACWP or missing performance data (e.g., BCWP with no BCWS or ACWP) should be rare If these anomalies occur they should be fully explained in the variance analysis portion of the report 4
EVM Findings from Recent Audits Many civil agency programs do not use product-oriented Work Breakdown Structures Schedules underpinning the EVM system are not meeting many best practices (detailed discussion to follow) IBRs are not occurring in a timely manner and are often not robust reviews Programs often rebaseline due to overly optimistic cost and schedule estimates EVM data anomalies are widespread and recurring Government program offices are not rejecting the EVM reports Format 5 variance analyses are too vague to be useful and do not address corrective actions EVM data are not being used to proactively manage the program Program managers doe not integrate EVM data with the risk management process Civil agencies do not have access to independent surveillance functions Government and contractor staff need additional training on EVM Contractors are not properly implementing their EVM systems 5
GAO Findings Related to Audit Report GAO-10-2 Information Technology: Agencies Need to Improve the Implementation and Use of Earned Value Techniques to Help Manage Major System Acquisitions October 2009
Audit Objectives Senator Thomas Carper, Chairman of Senate Government Affairs Subcommittee on Federal Financial Management, asked us to 1) Assess whether key agencies have appropriately established EVM policies. 2) Determine whether these agencies are adequately using earned value techniques to manage key system acquisitions. 3) Evaluate the earned value data of these selected investments to determine their cost and schedule performances. 7
Audit Scope & Selection Methodology The 8 selected agencies made up about 75% of planned federal IT expenditures (totaling $71 billion) in FY09. The 16 selected major IT investments from these agencies represent about $3.5 billion in total planned spending for development work in FY09. 8
Assessment of Agency EVM Policies None of the agencies policies were fully consistent with all 7 components of an effective EVM policy. Clear criteria for Agency implementing EVM on all major IT investments Compliance with the ANSI standard Standard structure for defining the work products Integrated baseline review Training requirements Rebaselining criteria System surveillance Agriculture Commerce Defense Homeland Security Justice NASA Transportation Veterans Affairs Key = The agency addressed all EVM practices in this policy area. = The agency addressed some EVM practices in this policy area. = The agency did not address any EVM practices in this policy area. Source: GAO analysis of agency data. 9
Assessment of Agency EVM Policies Criteria for implementing EVM 7 agencies had clear criteria, but VA s policy was open to interpretation Compliance with the ANSI standard 7 of 8 agencies required compliance, but Transportation s policy was vague. Standard structure for defining work products DOD was the only agency to meet this practice. 4 other agencies did not meet the practice while 3 partially met it. For example, NASA requires a standard WBS for mission but not IT projects. DHS and Justice have yet to standardize their product structures. Integrated Baseline Review (IBR) All 8 agencies required IT investments to conduct IBRs to ensure the baseline reflected full scope of work and resources as well as understood key risks. Training Requirements Commerce was the only agency to fully meet this practice for all personnel. All other agencies required training for project managers but not executives. Rebaselining 3 of 8 agencies fully met this practice. For instance, DOD, Justice, and Commerce s policies outlined acceptable reasons for rebaselining and required justification for doing so while the other agencies did not. System Surveillance All 8 agencies required ongoing EVM surveillance to ensure continued compliance with industry standards. 10
Assessment of Investment EVM Practices The GAO Cost Guide identifies 11 key EVM practices 1 that are implemented on acquisition programs of leading organizations. Program Management Area of Responsibility Establish a comprehensive EVM system EVM Practice Define the scope of effort using a WBS Identify who in the organization will perform the work Schedule the work Establish the labor and material required to do the work and authorize the budgets, including management reserve Determine objective measure of earned value Develop the performance measurement baseline Ensure that the EVM data are reliable Execute the work plan and record all costs Analyze EVM performance data and record variances Forecast estimates at completion Ensure that the program management team is using EVM data for decisions Take management action to mitigate risks Update the performance measurement baseline as changes occur 1 We did not review the two higher maturity steps associated with conducting an integrated cost-schedule risk analysis and comparing the EACs from the EVM data with the EAC from the risk analysis. 11
Assessment of Investment EVM Practices Agency Agriculture Program Farm Program Modernization Establishing a comprehensive EVM system Ensuring that data resulting from the EVM system are reliable Ensuring that the program management team is using earned value data for decisionmaking purposes Commerce Decennial Response Integration System Field Data Collection Automation Defense Air and Space Operations Center Weapon System Joint Tactical Radio System Handheld, Manpack, Small Form Fit Warfighter Information Network Tactical DHS Automated Commercial Environment Integrated Deepwater System Common Operational Picture Western Hemisphere Travel Initiative Justice Next Generation Identification NASA James Webb Space Telescope Juno Mars Science Laboratory Transportation En Route Automation Modernization Surveillance and Broadcast System Veterans Affairs Veterans Health Information Systems and Technology Architecture Foundations Modernization 12
Assessment of Investment EVM Practices Of the 16 programs, 3 fully implemented the practices essential to establishing a comprehensive EVM system and 13 partially implemented the practices. In most cases, programs had WBSs that defined work products to an appropriate level of detail and identified personnel responsible for delivering them. However, weaknesses in other practices were identified: 13 programs had project schedules that contained issues that undermined the quality of their performance baselines, including improper sequencing of activities, such as missing links between tasks; a lack of resources assigned to all activities; invalid critical paths; and the excessive use constraints. 9 programs did not adequately determine an objective measure of EV and develop the performance baseline. 4 programs did not define the scope of work using a WBS. 13
Assessment of Investment EVM Practices Of the 16 programs, 7 fully implemented the practices for ensuring the reliability of the earned value data, and 9 partially implemented them. The 7 programs that fully met the practices Conducted monthly reviews of EVM data with technical engineering staff and other key personnel to ensure the data are consistent with actual performance. Performed trend analysis to track program progress, cost, and schedule drivers, and developed EACs. The other 9 programs had processes in place to review the EVM data, identify and record variances, and forecast EACs. However, 5 of these programs did not adequately analyze EVM data and record variances 7 programs did not demonstrate they could adequately execute the work plan and record costs because they were unaware of the schedule weaknesses we found and did not have sufficient internal controls to improve the reliability of the data 2 programs could not adequately forecast EACs due to EVM data anomalies and schedule issues 14
Assessment of Investment EVM Practices Of the 16 programs, 9 fully implemented the practices for using earned value data to make decisions, 6 partially implemented them, and 1 did not. The 9 fully met programs integrated their EVM and risk management processes to support the program manager in making better decisions. Actively recorded risks associated with major variances from the EVM reports into the program s risk register. Used the data to analyze threat against remaining management reserve and to estimate the cost impact of these threats. The 6 programs that demonstrated limited capabilities included trend data in monthly program review briefings; however, many Had ad hoc processes for taking management action to address poor cost and schedule drivers that were not integrated with the program s risk management and Were not able to update the performance baseline as changes occurred because in several cases the original baseline was not appropriately validated The remaining program did not actively manage earned value performance trends or incorporate the data into program management reviews. 15
Assessment of Investment EVM Practices Inconsistent application of EVM across investments existed in part due to weaknesses found in agency policies and lack of enforcement of policies currently in place. Deficiencies in all three mgmt areas can be attributed to a lack of comprehensive training requirements. Agencies are not enforcing program EVM compliance at program-start, IBRs, and system surveillance. Until key EVM practices are fully implemented, selected programs face an increased risk that program managers cannot effectively optimize EVM as a management tool and reverse poor performance trends. 16
Assessment of Investment EV Performance Trends The earned value trends of the 16 programs indicate that most are currently experiencing cost overruns and schedule slippages. Exceeded cost targets by $275M. Have not completed $93M worth of work. Based on current contractor performance trends, 11 programs will likely experience cost overruns at completion of almost $1B with 2 of these programs accounting for 80% of this projection. In contrast, the contractor-estimate of cost overruns at completion was $470M with the assumption that efficiency would dramatically improve over what has been done to-date. 17
Assessment of Investment EV Performance Trends With timely and effective action taken by management, it is possible to reverse negative performance trends so that projected cost overruns can be reduced. To get such results, management at all levels (contractor, program office, executive) could be strengthened. For example, Obtaining EV data weekly (instead of monthly) so that they can make decisions with immediate greater impact. Elevating key risks to the program level and, if needed, executive level to ensure appropriate mitigation plans are in place and tracked to closure. 18
Recommendations We made 3 recommendations to the eight agencies: Modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report; Direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that were identified; and Direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the EVM data, to mitigate the potential cost and schedule overruns. The agencies generally agreed with GAO s findings and recommendations. 19