From: Government Accountability Office
Posted: Wednesday, July 26, 2006
July 17, 2006
The Honorable Sherwood Boehlert Chairman
The Honorable Bart Gordon Ranking Minority Member
Committee on Science
House of Representatives
Subject: NASA: Long-Term Commitment to and Investment in Space Exploration Program Requires More Knowledge
The National Aeronautics and Space Administration (NASA) plans to spend nearly $230 billion over the next two decades implementing the Vision for Space Exploration. In January 2006, NASA publicly released its Exploration Systems Architecture Study (ESAS), which is an effort to identify the best architecture and strategy to implement the President's 2004 Vision for Space Exploration (Vision).1 The cost estimate for implementing the ESAS through fiscal year 2011 exceeds $31 billion. The estimate through fiscal year 2018 is over $122 billion, and the estimate through fiscal year 2025 is nearly $230 billion.2 These estimates include the architecture, robotic precursor missions, supporting technologies, and funding needed to service the International Space Station (ISS).3 NASA plans to implement this architecture through a "go as you can afford to pay" approach, wherein lower-priority efforts would be deferred, descoped, or discontinued to allow NASA to stay within its available budget profile. This approach assumes NASA's budget will increase moderately to keep pace with inflation. Given the long-term fiscal imbalances that will challenge the entire federal government now and in the future, it would be prudent for NASA to establish a program that reduces the risk that significant additional funding, beyond moderate increases for inflation, will be required to execute the program.4 Government leaders will have to make difficult decisions to resolve such challenges, and the debate over the potential cost and the federal government's role in implementing the Vision are emblematic of the challenges the nation will need to resolve in the years ahead. Because of the significance of this investment, competing demands on the federal discretionary budget, and the importance of the success of NASA's exploration program to the future of U.S. human spaceflight, you requested that we assess (1) the extent to which NASA has identified the architecture and costs necessary to implement the Vision, (2) whether NASA's exploration architecture cost estimates fit within the agency's projected available budgets, and (3) the risks associated with NASA's acquisition strategy for the CEV project.
We presented our preliminary findings to your staff in May 2006. Because of your committee's interest in how NASA is implementing the Vision for Space Exploration, we are enclosing the full briefing that supported that May presentation with this report (see encl. II), along with a summary of our findings and conclusions. We are recommending that the NASA Administrator modify the current CEV acquisition strategy to ensure that the agency does not commit itself, and in turn the federal government, to a long-term contractual obligation prior to establishing a sound business case at the project's preliminary design review. In written comments, NASA nonconcurred with our recommendation and stated that it has the appropriate level of knowledge to proceed with its current acquisition strategy. As a result of its nonconcurrence, we are including as a matter for congressional consideration that the Congress should consider restricting NASA's appropriations and obligations for the CEV project to only the amount of funding necessary to successfully complete the project's preliminary design review.
The Vision includes plans to explore the moon, Mars, and beyond.5 The first step in implementing the Vision is to retire the space shuttle after completing assembly of the ISS by the end of the decade. NASA currently plans to retire the space shuttle in 2010, creating a potential gap in U.S. human spaceflight of up to 4 years before development of the CEV and the CLV is complete. Congress has voiced concern over the United States not having continuous access to space, and NASA has made it a priority to minimize the gap by accelerating the CEV project to have it in service as close to 2010 as possible. NASA's Exploration Systems Mission Directorate's (ESMD) Constellation program is responsible for the development of both the CEV and the CLV. NASA awarded concept development contracts for the CEV project to both Lockheed Martin and Northrop Grumman in July 2005 and plans to award a contract for design, development, production and sustainment in September 2006. That contract could extend through 2019. For the CLV, NASA plans to award a sole-source contract for the first stage of the CLV to ATK-Thiokol, the manufacturer of the Shuttle's Reusable Solid Rocket Motor, in October 2006. Also, the agency plans to award Pratt & Whitney Rocketdyne, the developer of the Space Shuttle Main Engine (SSME) and J-2 engines, a sole-source contract for development of the J-2X engine in November 2006. These contractors are currently planning their respective efforts under interim contract arrangements. NASA has started in-house preliminary design work on the CLV upper stage structures and avionics and plans to begin awarding competitive contracts for production of these items in May 2007.
Despite many successes in the exploration of space, such as landing the Pathfinder and Exploration Rovers on Mars, the loss of life, unsuccessful missions, and unforeseen cost overruns have recently increased the level of concern over the benefits of such exploration, particularly with regard to human spaceflight activities. NASA has had difficulty bringing a number of projects to completion, including several efforts to build a second generation of reusable human spaceflight vehicle to replace the space shuttle. NASA has attempted several expensive endeavors such as the National Aero-Space Plane, the X-33 and X-34, and the Space Launch Initiative, among others. While these endeavors have helped to advance scientific and technical knowledge, none have completed their objective of fielding a new reusable space vehicle. We estimate that these unsuccessful development efforts have cost approximately $4.8 billion since the 1980s. The high cost of these unsuccessful efforts and the potential costs of implementing the Vision make it important that NASA achieve success in its new exploration program.
Our past work has shown that developing a sound business case, based on matching requirements to available and reasonably expected resources before committing to a new product development effort, reduces risk and increases the likelihood of successful outcomes.6 At the heart of a business case is a knowledge-based approach to product development that is a best practice among leading commercial firms and successful government system developers. For a program to increase its chances of delivering a successful product, high levels of knowledge should be demonstrated before managers make significant program commitments. In essence, knowledge supplants risk over time. This building of knowledge can be described as three levels that should be attained over the course of the program:
(1) At program start, the customer's needs should match the developer's available resources in terms of availability of mature technologies, time, human capital, and funding.
(2) Midway through development, the product's design should be stable and demonstrate that it is capable of meeting performance requirements.
(3) By the time of the production decision, the product must be shown to be producible within cost, schedule, and quality targets, and have demonstrated its reliability.
Our work has shown that programs that have not attained the level of knowledge needed to support a sound business case have been plagued by cost overruns, schedule delays, decreased capability, and overall poor performance. With regard to NASA, we have reported that in some cases the agency's failure to define requirements adequately and develop realistic cost estimates—two key elements of a business case—resulted in projects costing more, taking longer, and achieving less than originally planned.
Although NASA is continuing to refine its exploration architecture cost estimates, the agency cannot at this time provide a firm estimate of what it will take to implement the architecture. The absence of firm cost estimates is mainly due to the fact that the program is in the early stages of its life cycle. According to NASA cost-estimating guidance, early life cycle phase estimates are generally based upon parametric models, which use data from projects with similar attributes to predict cost because there are usually many unknowns and actual cost or performance data are not available. NASA preliminarily identified the resources needed to implement the architecture as outlined in the architecture study primarily through the use of such models. NASA conducted a cost risk analysis of its preliminary estimates through fiscal year 2011. On the basis of this analysis and through the addition of programmatic reserves (20 percent on all development and 10 percent on all production costs), NASA is 65 percent confident that the actual cost of the program will either meet or be less than its estimate of $31.2 billion through fiscal year 2011. For the cost estimates for beyond 2011, when most of the cost risk for implementing the architecture will be realized, NASA has not applied a confidence level distinction. Since NASA released its preliminary estimates, the agency has continued to make architecture changes. For example, following the issuance of the architecture study, NASA conducted several analysis cycles during which various aspects of the architecture have evolved, such as the diameter of the CEV, the engine used to support the upper stage of the CLV, and the size of the Reusable Solid Rocket Booster on the CLV. While these changes, and others, are appropriate for this phase of the program, when concepts are still being developed, they leave the agency in the position of being unable to firmly identify program requirements and needed resources, which can also be expected at this phase of the program. According to NASA officials, once they receive more detailed contractor inputs, the agency will be able to produce higher-fidelity estimates of program cost. NASA plans to commit to a firm cost estimate at the preliminary design review (PDR) in 2008, when the program's requirements, design, and schedule will all be baselined.
NASA will be challenged to implement the architecture recommended in the study within its projected budget. Whether using the architecture study estimates of funds available or NASA's Fiscal Year 2007 Budget Submission for ESMD that was based on the architecture study cost estimates, there are years when NASA does not have sufficient funding to implement the architecture. Some yearly shortfalls exceed $1 billion, while in other years the funding available exceeds needed resources. NASA maintains that the architecture could be implemented within the projected available budgets through fiscal year 2011 when funding is considered cumulatively. In addition, NASA preliminarily projects multibillion-dollar shortfalls for ESMD in all fiscal years from 2014 to 2020, with an overall deficit through 2025 of over $18 billion. In the short term, NASA is attempting to address this problem within the Constellation program by redirecting funds to that program from other ESMD activities to provide a significant surplus for fiscal years 2006 and 2007 to cover projected shortfalls beginning in fiscal year 2009. In addition, the Constellation program has requested more funds than required for its projects in several early years to cover shortfalls in later years. For example, the Exploration Communication and Navigation Systems project within the Constellation program plans to roll over $56.2 million from the fiscal year 2007 budget to make up for budget shortfalls in fiscal years 2008, 2009, and 2010. NASA officials stated the identified budget phasing problem could worsen given that changes made to the exploration architecture following issuance of the study will likely add to the near-term development costs, where the funding is already constrained. In addition, NASA's estimates beyond 2010 are based upon a surplus of well over $1 billion in fiscal year 2011 due to the retirement of the space shuttle fleet in 2010. However, NASA officials said the costs for retiring the space shuttle and transitioning to the new program are not fully understood, and thus the expected surplus could be less than anticipated.
NASA's current acquisition strategy for the CEV places the project at risk of significant cost overruns, schedule delays, and performance shortfalls because it commits the government to a long-term product development effort before establishing a sound business case. NASA plans to award a contract for the design, development, production, and sustainment of the CEV in September 2006—before it has developed key elements of a sound business case, including well-defined requirements, a preliminary design, mature technology, and firm cost estimates. The period of performance for the contract scheduled for award in September 2006 will extend through at least 2014, with the possibility of extending through 2019. This contract will comprise all design, development, and test and evaluation activities, including production of ground and flight test articles and at least four operational CEVs. Although NASA is committing to a long-term contract, it will not have the elements of a sound business case in place until the project level PDR in fiscal year 2008. Awarding a contract for design, development, production, and sustainment of the project as NASA has planned places the CEV project at increased risk of cost growth, schedule delays, and performance shortfalls. At PDR, NASA will likely (a) have the increased knowledge necessary to develop a sound business case that includes high-fidelity, engineering-based estimates of life cycle cost for the CEV project, (b) be in a better position to commit the government to a long-term effort, and (c) have more certainty in advising Congress on required resources.
Implementing the Vision over the coming decades will require hundreds of billions of dollars and a sustained commitment from multiple Administrations and Congresses over the length of the program. The realistic identification of the resources needed to achieve the agency's short-term goals would provide support for such a sustained commitment over the long term. With a range of federal commitments binding the fiscal future of the United States, competition for resources within the federal government will only increase over the next several decades. Consequently, it is incumbent upon NASA to ensure that it is wisely investing its existing resources. As NASA begins to implement the Vision with several key acquisition decisions planned to occur this fall, it will be essential that the agency ensure that the investment decisions it is making are sound and are based upon high levels of knowledge. NASA should make the prudent decision now to ensure that it has attained the appropriate level of knowledge to support a sound business case before it commits to the project. However, under the current acquisition strategy for CEV, key knowledge—including well-defined requirements, a preliminary design, mature technology, and firm cost estimates—will not be known until over a year after the expected contract award date. Nevertheless, NASA plans to commit the government to a long-term contract. This approach increases the risk that the project will encounter significant cost overruns, schedule delays, and decreased capability. Given the nation's fiscal challenges and those that exist within NASA, the availability of significant additional resources to address such issues, should they occur, is unlikely. With the impending decisions pertaining to the CEV, NASA has the opportunity to establish a firm foundation for the entire Constellation program by ensuring that the appropriate level of knowledge is available before proceeding with its acquisition strategy and committing the government to a long-term design, development, and production effort.
Recommendation for Executive Action
Because of the importance of the CEV project to NASA's overall implementation of the Vision, NASA should focus on ensuring that its acquisition approach for the CEV project does not place the government at risk by committing to a long-term design and development effort without the knowledge needed to make wise investment decisions. We therefore recommend that the NASA Administrator modify the current CEV acquisition strategy to ensure that the agency does not commit itself, and in turn the federal government, to a long-term contractual obligation prior to demonstrating, through the establishment of a sound business case at the project's preliminary design review, that the project is affordable and executable.
Matter for Congressional Consideration
Based on its response to our report, it appears that NASA plans to proceed with its acquisition strategy for the CEV and award a long-term contract for the project, although it continues to lack sufficient knowledge and a sound business case for doing so. Congress is currently being asked to approve NASA's fiscal year 2007 funding request and will be asked to approve fiscal year 2008 and perhaps the fiscal year 2009 funding requests for the CEV project before NASA has demonstrated such knowledge and has provided evidence, based on that knowledge, that the project will be executable within existing and expected resources. In light of the fact that NASA plans to award the contract for the CEV in September 2006, Congress should consider restricting annual appropriations and limiting NASA's obligations for the CEV project to only the amount of funding necessary to support activities needed to successfully complete the project's preliminary design review.
Agency Comments and Our Evaluation
In written comments on a draft of this report (see encl. I), NASA nonconcurred with our recommendation that it modify the current CEV acquisition strategy to ensure that the agency does not commit itself, and in turn the federal government, to a longterm contractual commitment prior to establishing a sound business case at the project's preliminary design review. NASA stated that it has the appropriate level of knowledge to proceed with its acquisition plan to "down select" to a single Crew Exploration Vehicle prime contractor in September 2006. NASA added that it is maximizing competition by soliciting from industry a development, production, and management approach with an emphasis on life cycle cost. In the area of technology maturity, NASA stated that it has a plan and process in place to address the Thermal Protection and Landing subsystems technology risks through in-house development work and collaboration with the prime contractor. NASA also noted that during its design, development, and test and evaluation effort, the agency will be using an enditem award fee, which would make all award fees subject to a final evaluation to determine how well the product met requirements, including cost and schedule.
The CEV acquisition strategy is not knowledge-based in that it calls for maturing technologies, designing systems, and preparing for initial production concurrently— an approach that our work has shown carries the increased risk of cost and schedule overruns and decreased technical capability. Therefore, we disagree with NASA's statement that it has the appropriate level of knowledge to proceed with its current acquisition strategy and award a long-term contract for the project prior to obtaining sufficient knowledge. Specifically:
It is important to note that Congress will continue to be asked to make funding commitments in advance of CEV project events that would demonstrate that the project has the knowledge necessary to support a sound business case. Specifically, NASA's funding request for fiscal years 2007 and 2008 are scheduled to be approved before the CEV holds its preliminary design review. Since the preliminary design review is currently scheduled for March 2008, this may also be the case for fiscal year 2009. Congress should safeguard against a situation in which contractual and budget decisions could hinder its ability to tie further investments in the CEV project to demonstrated progress at the preliminary design review. As such, we have included a matter for congressional consideration.
We also received technical comments from NASA, which have been addressed in the report, as appropriate.
Scope and Methodology
To assess the extent to which NASA has identified the architecture and costs necessary to implement the Vision and whether NASA's exploration architecture cost estimates fit within the agency's projected available budgets, we reviewed and analyzed NASA's Exploration Systems Architecture Study, fiscal year 2007 budget request, ground rules and assumptions provided from the Constellation program to project level management estimators to perform the bottom up review, guidance for use in preparing the fiscal year 2008 budget request, NASA cost-estimating guidance in the NASA Cost Estimating Handbook, and congressional hearings and testimonies pertaining to NASA and the Vision. We also conducted interviews with NASA headquarters officials from the Cost Analysis Division, the Exploration Systems Mission Directorate, and Constellation program officials, Constellation program and CEV project officials at Johnson Space Center; CLV project officials at Marshall Space Flight Center; and cost analysts from the Kennedy Space Center. During these interviews, we discussed the methodologies used in preparing the ESAS and subsequent cost estimates, architecture changes after the ESAS and the trades being considered, budgeting issues, and procurement strategies and activities.
To assess the risks associated with NASA's acquisition strategy for the CEV project, we reviewed and analyzed CEV project documentation, including draft project plans, draft requirements documents, technology development plans, documentation included in the contract request for proposals, and past NASA human spaceflight acquisition programs. We compared NASA's plans for the CEV with criteria contained in GAO best practices work on systems acquisition. We also conducted interviews with NASA headquarters officials from the Exploration Systems Mission Directorate and Constellation Systems officials, Constellation program and CEV project officials at Johnson Space Center, and CLV project officials at Marshall Space Flight Center. We conducted our work from January 2006 to May 2006 in accordance with generally accepted government auditing standards.
As agreed with your offices, unless you announce its contents earlier, we will not distribute this report further until 10 days from its date. At that time, we will send copies of the report to NASA's Administrator and interested congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO's Web site at http://www.gao.gov. Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or firstname.lastname@example.org. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Principal contributors to this report were James L. Morrison, Assistant Director; Rick Cederholm; Shelby S. Oakley; Guisseli Reyes; Sylvia Schatz; and
John S. Warren, Jr.
Acquisition and Sourcing Management
1 The ESAS architecture supports the development of a new Crew Exploration Vehicle (CEV), Crew Launch Vehicle (CLV), a Cargo Launch Vehicle (CaLV), and other supporting systems. The architecture also calls for various Research and Technology (R&T) and Robotic Lunar Exploration Program (RLEP) projects.
2 All cost estimates related to the Vision are reported as inflated ("real year") dollars.
3 NASA's cost estimate through 2011—$31.2 billion—included the costs of the R&T and RLEP projects needed to support the architecture. Its estimate for the first lunar landing—$104 billion—did not include $18 billion in funding for R&T and RLEP projects. To ensure consistency, the estimates for 2018 and 2025 are presented with R&T and RLEP funding included.
4 GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: Feb. 2005); 21st Century: Addressing Long-Term Fiscal Challenges Must Include a Reexamination of Mandatory Spending, GAO-06-456T (Washington, D.C.: Feb. 15, 2006); and Highlights of a GAO Forum: The Long-Term Fiscal Challenge, GAO-05-282SP (Washington, D.C.: Feb. 1, 2005).
5 The Vision includes a return to the moon that is intended ultimately to enable future exploration of Mars and other destinations. To accomplish this, NASA initially plans to (1) complete its work on the International Space Station by 2010, fulfilling its commitment to 15 international partner countries; (2) begin developing a new manned exploration vehicle to replace the space shuttle; and (3) return to the moon no later than 2020 in preparation for future, more ambitious missions.
6 Examples of our best practices reports include GAO, Best Practices: Using a Knowledge-Based Approach to Improve Weapon Acquisition, GAO-04-386SP (Washington, DC.: Jan. 2004); Space Acquisitions: Committing Prematurely to the Transformational Satellite Program Elevates Risks for Poor Cost, Schedule, and Performance Outcomes, GAO-04-71R (Washington, D.C.: Dec. 4, 2003); Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: Jul. 15, 2002); and Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, DC.: Mar. 8, 2001).
// end //