NASA has released the executive summary of Booz Allen Hamilton’s independent analysis of the Space Launch System (SLS), Multi-purpose Crew Vehicle (MPCV) and the required ground systems. The review says that near-term cost estimates are fairly reasonable, but that longer-term cost figures are overly optimistic and that programs reserves are insufficient. A key section of the executive summary is below, with my emphasis added:
In general, the estimates prepared by SLS, MPCV, and 21CGS are consistent with Analysis of Alternative (AoA) level estimates and are reasonable point estimates for budget planning in the near-term 3-5 year budget horizon. They are serviceable in that they represent the basis to build upon for future life-cycle cost estimates of the quality required for long-term budget formulation and the development of program baselines. None of the estimates reviewed by the ICA Team support establishment of long-term budgets or detailed baselines consistent with NPR 7120.5 requirements. They are, however, reasonable AoA estimates appropriate for supporting trade studies and comparative analyses. All three Program estimates assume large, unsubstantiated, future cost efficiencies leading to the impression that they are optimistic. A scenario-based risk assessment, which excludes cost estimating uncertainty and unknown-unknown risks (historically major sources of cost and schedule growth), reveals all three Programs’ reserves are insufficient.
The key findings and recommendations are reproduced after the break.
Below are the seven key findings common to all three programs:
Finding: The Programs’ estimates are serviceable and can be used for near-term budget planning in the current 3- to 5-year budget horizon. Beyond this horizon, the inclusion of large expected cost savings in the estimates, the beginning of development activities, and the potential for significant risk events decreases the ICA Team’s confidence in the estimates.
Recommendation: The ICA Team recommends that NASA initiate full Program life-cycle cost estimates (LCCEs) for SLS, MPCV, and 21CGS immediately upon approval of SLS architecture. Further, the ICA Team recommends that an Independent Cost Estimate be conducted at the next program milestone as a cross-check to the Program estimates.
Finding: The BOEs provided by the Programs are not fully traceable or documented. An independent organization could not replicate Program estimates using the data sets provided by SLS, MPCV, or 21CGS without additional explanation from Program staff.
Recommendation: The ICA Team recommends that HEO/ESD use the occasion of a selection of a new SLS architecture to establish a common practice across Programs for generating cost and schedule estimates; establish documentation standards for BOEs; and create and disseminate BOE, cost, and schedule estimate templates to Programs.
Finding: There are many instances of unjustified cost reductions in the Program estimates. This exposes the Programs to cost risk and undermines the credibility of the estimate. Cost reductions were generally observed in either of two categories: scope reductions where the removed work will likely be required, or the application of anticipated efficiencies (production, competition, etc.) that NASA has not historically achieved. In some cases the efficiencies leading to cost reductions are not explicitly identified. Both of these categories lower the estimates below what historical data suggests and indicate that the estimates are optimistic.
Recommendation: The ICA Team recommends HEO/Exploration Systems Development (ESD) require that adjustments exceeding +/- 5% from historical data provide written justifications for such adjustments and take countervailing reserve positions for items where justifications are not sufficient. Where the source of cost reductions are not identified, NASA should describe the rationale for such reductions and develop a management plan to increase the probability that NASA achieves these efficiencies.
Finding: Programs lack sufficient reserves to cover their Protect Scenarios. Program reserves were applied independent of the level of cost and schedule risk. Quantitative risk/sensitivity analyses have not been performed on any estimates.
Recommendation: The ICA Team recommends quantitative risk/sensitivity analyses be performed on all three Programs in conjunction with their full-scale life-cycle cost estimates (recommended above) and that reserve positions be established based on risk analysis findings.
Finding: There is little observed evidence of formal cross-Program coordination or integrated schedule development, although the Programs do interact on a working level. Lack of formal integration among the Programs diminishes each program manager’s ability to manage cost and schedule risk proactively.
Recommendation: The ICA Team recommends HEO/ESD enhance PMO functionality by assigning a core group of dedicated staff, working independent of the individual Programs, to coordinate and integrate schedules, milestones, technical reviews, and cross-program risks.
Finding: Individual Program cost and schedule estimates are managed separately and are not in alignment. Lack of cost and schedule alignment makes it difficult to estimate the cost impact of schedule slips and vice-versa. Non-integrated cost and schedule estimates are sufficient for trade studies, but do not facilitate ongoing baseline management.
Recommendation: The ICA Team recommends that HEO/ESD establish a common WBS/CES as a foundation for future integration of cost and schedule.
Finding: Program estimates were shaped to fit within an anticipated budget profile. While this is common practice, it is not consistent with GAO cost estimating best practices. Shifting costs to later years to fit upcoming budget caps decreases each Program’s availability of funds for risk reduction, technology maturation, and exposes the Programs to out-year cost growth.
Recommendation: The ICA Team recommends that time-phased results from the full LCCE be used to determine whether or not the AoA estimates, phased to fit into a budget profile, are executable. If not, the ICA Team recommends Programs identify specific plans to move work between years to fit into the budget cap.
Read the full Executive Summary.