From: Government Accountability Office
Posted: Wednesday, June 20, 2001
This report discusses the National Aeronautics and Space Administration's (NASA) contract with Boeing Reusable Space Systems to build the now-canceled follow-on propulsion module for the International Space Station. GAO found that the initial propulsion module project did not meet performance, cost, and schedule goals largely because NASA proceeded with Boeing's proposal without following fundamental processes involving project planning and execution. Once it was determined that Boeing's proposal was inadequate, NASA began to assess alternatives to the Boeing-proposed propulsion module. The assessment team defined mission success criteria, identified key design assumptions, and performed comparative analysis on competing designs. On the basis of its analyses, the team recommended a follow-on design. NASA acknowledged that its initial approach to developing a propulsion module was inadequate and contributed to the project's unsuccessful conclusion. NASA officials sought to learn lessons from the project in order to avoid similar problems in managing future programs.
Report to Congressional Requesters
United States General Accounting Office
June 2001 SPACE STATION Inadequate Planning and Design Led to Propulsion Module Project Failure
Page i GAO-01-633 Space Station Letter 1
Appendix I Comments From the National Aeronautics and Space Administration 15.
Appendix II Staff Acknowledgments 17
Figure 1: Diagram of Node X Option 11 Contents
Page 1 GAO-01-633 Space Station
June 20, 2001 The Honorable Ernest F. Hollings Chairman The Honorable John McCain Ranking Minority Member Committee on Commerce, Science, and Transportation United States Senate
The Honorable Sherwood L. Boehlert Chairman The Honorable Ralph M. Hall Ranking Minority Member Committee on Science House of Representatives
The National Aeronautics and Space Administration (NASA) faces many challenges in developing and supporting the International Space Station. These challenges, such as Russian difficulty in completing its components on schedule, led NASA to pursue development of a U. S. propulsion capability for the space station to serve as an alternative to the planned Russian capability. In 1998, NASA accepted a proposal from Boeing Reusable Space Systems for a U. S. propulsion module. NASA's initial effort to develop this module was not successful in meeting the program's performance, cost, and schedule goals. The effort failed to produce a design that met mission requirements, increased its estimated cost by $265 million (from $479 to $744 million), and slipped its schedule by about 2 years. NASA eventually canceled the program and initiated a follow-on effort.
Although the follow-on propulsion module has since been terminated due to projected cost increases in the space station as a whole, you asked that we analyze the initial propulsion module project to help NASA avoid similar problems in the future. Specifically, we assessed NASA's processes for program planning and ensuring that the proposed design met technical requirements. In addition, this report describes NASA's evaluation of alternative approaches for the follow-on propulsion module effort and the results of NASA's lessons learned studies from the failed initial program.
United States General Accounting Office Washington, DC 20548
Page 2 GAO-01-633 Space Station
We have reported separately on NASA's decision to develop the initial propulsion module under the Boeing space station prime contract. 1 The initial propulsion module project did not meet performance, cost, and schedule goals largely because NASA proceeded with Boeing's proposal without following fundamental processes involving project planning and execution. NASA officials stated that, had these processes been followed, they would have determined earlier in the program that the Boeing proposal would not meet project goals. For example, NASA did not complete a project plan or develop sufficient information in areas such as systems analysis and risk management to guide the program. Having such basic information is fundamental to sound project management. In addition, Boeing's design was accepted and implemented before the propulsion module's detailed technical requirements were fully established. NASA later found that the design was not as mature as anticipated and that it required substantial changes. This led to significant delays, cost increases, and ultimately, project cancellation.
In May 2000, NASA began to assess alternatives to the Boeing-proposed propulsion module. The assessment team defined mission success criteria, identified key design assumptions, and performed comparative analyses on competing designs. Based on its analyses, the team recommended a follow-on design. According to NASA officials, this effort brought early analytical rigor to requirements definition, which NASA had failed to do in the initial project.
NASA acknowledged that its initial approach to developing a propulsion module was inadequate and contributed to the project's unsuccessful conclusion. NASA officials performed lessons learned efforts on the project in general and on one specific component-on-orbit fuel transfer-hoping to avoid similar problems in managing future programs. In all cases, NASA concluded that the lack of an early systems analysis contributed to project failure. Regarding the failed attempt to design an on-orbit fuel transfer component into the propulsion module, NASA cited difficulty in establishing requirements, estimating cost and schedule, and providing adequate resources.
1 International Space Station Propulsion Module Procurement Process (GAO- 01-576R, Apr. 26, 2001).Results in Brief
Page 3 GAO-01-633 Space Station
NASA commented on a draft of this report and agreed with our findings. We are not making recommendations at this time because NASA has canceled its effort to develop a propulsion module and is recognizing the need to avoid similar problems in the future by identifying lessons learned.
NASA and its international partners (Japan, Canada, the European Space Agency, and Russia) are building the space station as a permanently orbiting laboratory to conduct materials and life sciences research and earth observation and provide for commercial utilization and related uses under nearly weightless conditions. Each partner is providing station hardware and crew members and each is expected to share operating costs and use of the station.
Russia became a partner in 1993. As a partner, Russia agreed to provide hardware, such as the Service Module to provide station propulsion, supply vehicles, and related launch services throughout the station's life. However, Russia's funding problems delayed the launch of the Service Module by more than 2 years and raised questions about Russia's ability to support the station during and after assembly.
Shortly after Russia came into the program, NASA began studying ways to provide the required propulsion using existing designs and hardware. Later, in response to continuing problems in the Russian space program, such as declining launch rates and funding shortages, NASA initiated additional studies at the Marshall Space Flight Center in Alabama. In the spring of 1995, it focused on satisfying the space station's command and control and propulsion requirements. In 1996, Marshall proposed building a propulsion module in-house, and in 1997, NASA considered using existing Russian hardware to provide the needed propulsion.
In 1997, in response to continuing concerns over Russia's ability to fulfill its station commitments, NASA developed a contingency plan, which included a strategy to mitigate the risk of both further delay on the Service Module propulsion capabilities and Russia not being able to meet station propulsion needs. Key elements of the plan involved developing an interim control module for near-term needs and a propulsion module to provide a permanent U. S. capability.
Page 4 GAO-01-633 Space Station
In 1998, Boeing Reusable Space Systems proposed a propulsion module concept that was to rely heavily on existing shuttle hardware, provide for on-orbit refueling, and cost about $330 million. 2 This proposal coincided with renewed concern that Russia would not be able to fulfil its commitment to provide station propulsion capability. NASA decided to move ahead with the proposal based on a strategy that included refining the design during subsequent requirements and design reviews. It adopted this strategy based on its assumption that a propulsion module would be required by early 2002 if Russia failed to launch the Service Module. In July 2000, Russia successfully launched the Service Module, thus mitigating NASA's immediate concern that the space station would not have adequate propulsion capability. However, the agency still proceeded with the propulsion module project because of long-term concerns with Russia's ability to fulfil its commitments.
NASA proceeded with Boeing's proposal without following fundamental processes involving project planning and execution. Specifically, project management never finalized its project plan or operational concept and did not receive timely approval for its risk management plan. The design ultimately required substantial changes.
To prudently manage the project, NASA should have prepared and completed a number of planning documents and established baseline goals. 3 Specifically, NASA did not do the following:
Complete a project plan. 4 Documented project plans help to define realistic time frames for system acquisition and development and identify responsibilities for key tasks, resources, and performance measures. Without them, there is no yardstick by which to measure the progress of the developmental effort.
2 This initial estimate did not include the cost for on-orbit fuel transfer or costs associated with integrating the module into the station. In February 1999, the estimate was revised to $479 million to include all programmatic costs.
3 NASA Procedures and Guidelines 7120.5A. This document establishes the management system for processes, requirements, and responsibilities for implementing NASA programs and projects. This management system governs the formulation, approval, implementation, and evaluation of all agency programs and projects.
4 A draft project plan was in the process of being finalized when the project was canceled. Basic Project Management and Requirements Principles Not Followed Propulsion Module Project Planning Was Inadequate
Page 5 GAO-01-633 Space Station
Fully develop a concept of operations document. This document describes the range of operational scenarios under which project hardware will have to function. This document is necessary to define requirements, operational plans, and associated training. The project began with a rudimentary concept that was continually refined during the course of the project.
Complete an approved risk management plan in a timely manner. 5 A formal risk management plan helps management identify, assess, and document risk associated with the cost, resource, schedule, and technical aspects of the project. Without such a plan, organizations do not have a disciplined means to predict and mitigate risks.
Develop realistic cost and schedule estimates for the life of the project. NASA guidance states that life-cycle cost estimates shall be developed as a part of the project's planning activity.
Because of its concerns that Russia would be unable to provide space station propulsion capability, NASA approached the effort with a sense of urgency. Its analysis indicated that the U. S. propulsion capability would be needed by early 2002 if Russia did not meet its commitment. Given the initial estimate of the time that would be needed to develop and launch the propulsion module, NASA believed it was necessary to expedite the project. As a result, NASA chose to simultaneously plan and execute the project, thereby inhibiting use of fundamental planning documents during project formulation.
According to NASA officials, the absence of approved planning documentation and the urgency NASA perceived in executing the project made it difficult to effectively guide the project or measure its progress. For example, roles and responsibilities continued to change, impeding the flow of information. In addition, the absence of accurate technical, cost, and schedule estimates early in the project made it difficult for NASA to track cost variances. As a result, NASA officials told us that the estimated $265 million cost increase announced just before the program was suspended came as a surprise. They also stated that, had more analytical
5 While a risk management plan was needed early in the project, it was not formally approved until 3 months before the project ended. However, officials did provide evidence of risk identification, assessment, and mitigation strategies prior to the plan's approval. Project officials recognized that the risk management process was incomplete and that the possibility existed that an unnoticed risk could arise affecting performance, cost, and schedule. Some of the steps NASA undertook to mitigate identified risks included establishing assessment teams, imposing additional testing, and canceling high-risk development activity.
Page 6 GAO-01-633 Space Station
rigor been applied, they would have determined earlier in the program that the Boeing proposal would not meet project goals.
This procurement strategy also caused NASA to purchase long-lead items before the project's requirements, concept of operations, and costs were fully understood. According to NASA, prior to the decision to cancel the project, it had obligated about $40 million for the purchase of various longlead items. Some of these items could be used on the space station or other NASA projects. However, other items were unique to the propulsion module project.
Our findings on lapses in project planning are consistent with results from a NASA independent assessment team, which reviewed the propulsion module project between September 1999 and March 2000. This team concluded that the project was at high risk due, in part, to the fact that these critical project management processes were not followed. Specifically, the team concluded that (1) the project would not be ready to proceed through the design reviews until the project plan was fully developed and approved, (2) a well-integrated risk management program was not in place, and (3) the project could not be completed within the budget or achieve its planned delivery date.
NASA proceeded to implement Boeing's proposal before it determined whether the design would fully meet the project's technical requirements. The following top-level requirements were established at the beginning of the program:
Provide reboost and attitude control capability.Conform to NASA safety provisions. Even though the top-level requirements provided a framework to guide propulsion module development, NASA's reviews of the module's detailed technical requirements identified major concerns. For example, an April
Provide up to 50 percent of total on-orbit space station propulsion needs. 6 Provide 12-year on-orbit life.
Maintain orbiter compatibility and transfer capability (pressurized transfer tunnel for crew and supplies).
Provide capability to be launched and returned by the shuttle.
6 According to NASA, under this design concept a second module would be required to provide a fully independent U. S. propulsion capability. NASA Reviews Identified Deficiencies in the Propulsion Module Design
Page 7 GAO-01-633 Space Station
1999 systems requirements review found that NASA did not have detailed analyses to quantify the amount of propulsion capability that would be required. NASA space station program personnel later determined the definition of what propulsive capability was required; however, this definition was not available until a few months before the initial design review and could not, therefore, be used to judge the design's suitability. Subsequent reviews found deficiencies with the design elements of the module itself.
Although technical requirements were never finalized and continued to change, NASA accepted and began to implement Boeing's design. Typically, technical requirements are determined prior to selecting a design to ensure that it can satisfy established technical and safety needs. NASA accepted Boeing's proposed design and began to implement it because the agency believed (1) the design was stable and mature because some of the proposed hardware had been used on the space shuttle and (2) costs were essentially fixed because the required development activities were understood and would not change.
As NASA implemented the design-establishing the organization for and responsibilities of the project office, purchasing long-lead items, etc.- it discovered a number of unexpected technical complexities and other obstacles in the design. These problems put into question the ability of the design to meet the technical requirements, as indicated in the following examples.
A central requirement was for the propulsion module to be refueled while in orbit. NASA began to question Boeing's ability to incorporate on-orbit fuel transfer into the design, citing significant cost, safety, operational, and system design issues. Ultimately, NASA eliminated this requirement and reduced the module's on-orbit life expectancy from 12 to 6 years. Eliminating these requirements meant that the propulsion module had to return to earth for refueling; the concept of returnability had not been fully analyzed; thus, a new design team had to be established to assess these impacts.
The design also proposed a tunnel diameter that proved too small to accommodate crew operations and did not meet space station minimum diameter requirements. In addition to crew passage, the tunnel served as the primary path for equipment/ supply movement from the shuttle to the station. The tunnel size was later increased based on NASA's concerns.
The design made extensive use of existing shuttle flight hardware that had not been designed for a 12-year application, and Boeing assumed NASA would accept the hardware based on prior shuttle experience. However,
Page 8 GAO-01-633 Space Station
NASA assessed the hardware and found that much of it could not meet a 12- year life requirement. In addition, the development specification did not fully address testing requirements because Boeing assumed a simplified level of testing. However, testing requirements were later expanded.
The propulsion module project failed to successfully complete its preliminary design review in December 1999, despite the fact that it had been considered a mature design. 7 The review concluded, in part, that the initial propulsion system design did not meet the space station and space shuttle safety requirements and that another review of propulsion related issues was needed. In March 2000, NASA's independent assessment team concluded that the design was not mature, requirements were not adequately defined, and major design impacts were likely.
The process by which NASA and Boeing attempted to execute the project resulted in design changes, added effort, schedule slippage, and purchase of long-lead items before the design was fully understood. As a result, the project's total cost estimate increased significantly. In February 1999, Boeing estimated that total program cost would be $479 million and maintained that estimate until April 2000. At that time, Boeing increased its estimate by $265 million-from $479 million to $744 million. 8 Over this period, the scheduled launch date slipped by almost 2 years-from August 2002 to July 2004. Based on Boeing's revised estimates, NASA began to question the project's viability, and in July 2000 it informed Boeing that it would not authorize any additional work on the project. 9
7 The preliminary design review is the project's initial formal review to establish a design baseline. This is followed by the critical design review, after which the design is ready for manufacture, assembly, and integration of subsystems.
8 The cost estimate of $744 million did not include any contingency funding.
9 At the time NASA decided to stop work on the propulsion module effort, NASA and Boeing had not negotiated the final terms of the development contract. Boeing was expected to submit a proposal by the end of May 2000, but the module design continued to change significantly and the contractor was unable to meet that target date. When the project was stopped, NASA had obligated about $135 million for the development effort.
Page 9 GAO-01-633 Space Station
Prior to abandoning its original propulsion module design, NASA established an Alternative Propulsion Module Assessment Team in May 2000 to review design concepts for their potential to meet the space station program's propulsion requirements. According to NASA officials, this effort brought early analytical rigor to requirements definition, which NASA had failed to do in the initial project. During the preliminary phase of the assessment, team members considered many diverse options. These options varied in design factors such as module location, number of propulsion elements, and propellant systems. Each option also had to meet the basic top-level requirements. Specifically, the alternative propulsion module had to provide space station attitude and orbit maneuver control, be located on the U. S. segment of the station, leave two ports available for other vehicles to dock to the station, meet space station safety requirements, and initially provide 50 percent of the space station's propulsion needs. In addition, the design had to be adaptable to eventually provide 100 percent of the station's propulsion needs.
The assessment team identified five potential concepts, including two modified versions of Boeing's baseline propulsion module design; the Z1 truss option, which attached to the station's truss system; the split element option with separate propulsion and avionics elements; and the Node X option that had the propulsion elements attached to the Node 1 structural test article. 10 The team designated a subteam to refine each option's design. In addition, the subteams consulted with a cost assessment group to develop cost estimates for each option. The cost assessment group considered both initial capability costs, such as development and integration, and additional life-cycle cost elements, such as shuttle launches, labor, and spare parts.
Using the subteams? analyses, the assessment team ranked the propulsion module alternatives in three categories-programmatic (composed of schedule, cost, and risk), technical (including safety, design, and performance), and integration issues (such as International Space Station and shuttle impacts and logistics issues). The team weighted programmatic issues the highest at 60 percent and technical and integration issues at 20 percent each. NASA officials told us that these
10 The structural test article was built to undergo various pressure and leak tests to support Node 1 construction. Node 1 is a pressurized element that serves as the connecting passageway for the space station's other modules, and it was the first U. S. station element launched. Follow-on Propulsion Module Effort Included Comparative Analyses of Alternative Designs
Page 10 GAO-01-633 Space Station
weightings, typical for this kind of analysis, were approved by the space station program management.
The assessment team concluded that the Z1 truss option was the best choice. This option did not require the construction of a pressurized element and was estimated to cost $515 million to develop. The next best alternative was the Node X option, with an estimated cost of $700 million to develop. According to NASA, this option was already well understood because Boeing had already integrated a similar structure, Node 1, into the space station.
Although the assessment team found the Z1 truss option superior, it recommended a follow-on study because issues associated with this option's integration into the space station were not well understood. Consequently, in July 2000, a joint NASA and Boeing integration evaluation team examined integration risks and identified possible design improvements for the Z1 truss and Node X options. NASA believed Boeing's involvement was important because as the prime contractor, it would be responsible for integrating the alternative propulsion module into the space station.
The methodology that the integration evaluation team used was similar to that used by NASA's assessment team in reviewing the propulsion module options. The integration team designated individual teams to evaluate the ZI truss and Node X options from various functional perspectives, such as power; structures and mechanisms; guidance, navigation, and control; and contamination. The functional teams developed criteria for their particular discipline and evaluated the two options accordingly. For example, the structures and mechanisms team evaluated the two options for peak loads and structural fatigue, and the power team for average and peak power consumption. Based on its evaluation, each team recommended a preferred option. The integration team's project manager then led an effort to compile and analyze the functional teams? recommendations.
Based on this analysis, the team selected the Node X option, which had the highest overall mission suitability and lowest integration risk. In contrast, the Z1 truss option created structural stress, station controllability, and propellant inefficiency issues. The integration team then concluded that Node X was the preferred choice as a follow-on effort to the initial propulsion module project. Figure 1 depicts the Node X propulsion module configuration.
Page 11 GAO-01-633 Space Station
Figure 1: Diagram of Node X Option
Source: Marshall Space Flight Center. The cost assessment group incorporated the results of the integration evaluation team into a new cost analysis for the ZI truss and Node X options. According to the new cost analysis, the Z1 truss option's integration issues increased its estimated cost to $729 million. In contrast, the cost estimate for the Node X option decreased to $675 million, primarily because the outfitting costs for the structural test article were lower than expected.
NASA accepted the integration team's findings and issued a request for proposal on the Node X option in January 2001. However, 2 months later, NASA canceled the follow-on effort because of cost increases in the space station program as a whole. In addition, NASA believed that the risk of Russian nonperformance was reduced because of the Service Module's deployment.
Page 12 GAO-01-633 Space Station
NASA acknowledged that problems with the management of the initial propulsion module project contributed to its unsuccessful conclusion, and it is undertaking lessons learned efforts to help avoid similar problems in managing future programs. These assessments include top down and systems engineering reviews at the Marshall Space Flight Center and an assessment by an engineer at the Johnson Space Center in Texas related specifically to the on-orbit fuel transfer component of the module.
According to NASA officials, drafts of the Marshall assessments identified the lack of early systems analysis and good teamwork as contributing to project failure. For example, key components of the design-use of existing hardware, on-orbit fuel transfer, and tunnel size-were never tested for feasibility, partly because Boeing believed that NASA had fully accepted the assumptions inherent in its design. Later, when these assumptions became invalidated or retracted, it became apparent that the original concept was no longer feasible.
The top down assessment also cited a lack of cooperation between NASA and Boeing as inhibiting the timely completion of required design and risk management analyses. In many cases, when NASA and Boeing teams tried to work together, they became confrontational and nonproductive. It concluded, in part, that, in the future, NASA should ensure that (1) early planning documents define what roles the various project teams have and how they interact, (2) government and contractor counterparts are established to encourage collaboration, and (3) NASA and contractor management monitor the interaction of the overall project team and intercede if conflicts interfere with the project's success.
Another lessons learned assessment was performed on the on-orbit fuel transfer component of the Boeing proposal, a basic requirement of the proposed design. This assessment concluded that, while some project communication was good, internal contractor communication on this part was less than desirable. For example, center-to-center communication was aided by daily conversations between the on-orbit fuel transfer and propulsion module project managers. However, contractor participation in working group activities was not supported. The assessment also cited NASA's lack of systems analysis early in the program and its difficulty in establishing requirements, estimating cost and schedule, and providing human capital resources as contributing to the on-orbit fuel transfer project failure. NASA Recently
Identified Lessons Learned to Apply to Future Programs
Page 13 GAO-01-633 Space Station
The themes cited in NASA's propulsion module project lessons learned studies are consistent with those cited in previous program failure assessments. In December 2000, NASA issued a report synthesizing the findings and recommendations from earlier reports, arriving at five themes it considered necessary for sound project management. The five themes were developing and supporting exceptional people and teams, delivering advanced technology, understanding and controlling risk, ensuring formulation rigor and implementation discipline, and improving communication. 11
In commenting on a draft of this report, NASA stated that while it was in agreement with the findings of the report, the project's urgency necessitated its management approach. However, NASA acknowledges that its project execution could have been improved and that it will now strive to apply lessons learned from the propulsion module project experience.
Even though NASA perceived a schedule urgency in starting and completing the project, it should have followed sound management practices. The early analytical rigor NASA was applying to the follow-on propulsion module effort would have served the agency well in its execution of the initial project.
To assess the adequacy of project planning, we reviewed, analyzed, and compared internal NASA guidance and project planning documents. We also discussed planning requirements with cognizant project officials and independent assessment team officials to obtain their views.
To assess the extent to which NASA had defined the technical requirements for the propulsion module, we reviewed the results of requirements meetings and approved requirements lists to gain an understanding of the evolution of requirements determinations. We also held discussions with NASA and Boeing officials to obtain their perspectives on the validity of the technical requirements, as well as reasons for requirements changes over the course of the program.
11 Enhancing Mission Success -A Framework for the Future, a report by the NASA Chief Engineer and the NASA Integrated Action Team (Dec. 2000). Agency CommentsScope and Methodology
Page 14 GAO-01-633 Space Station
To describe NASA's process for reviewing alternative designs, we reviewed NASA briefing materials and other products related to the establishment of the Alternative Propulsion Module Assessment Team and others. We also discussed the teams? charter, methodology, and results with team members and other cognizant officials.
To describe lessons learned by NASA from the initial program, we reviewed the results of NASA's efforts and discussed their significance with cognizant officials.
We conducted our review from July 2000 to April 2001 in accordance with generally accepted government auditing standards.
Unless you publicly announce its contents earlier, we plan no further distribution of this report until 10 days from its issue date. At that time, we will send copies to the NASA Administrator; the Director, Office of Management and Budget; and other interested parties. We will also make copies available to others on request.
Please contact me at (202) 512-4841 if you or your staff have any questions about this report. Other key contributors to this report are acknowledged in appendix II.
Allen Li Director, Acquisition and Sourcing Management
Appendix I: Comments From the National Aeronautics and Space Administration
Page 15 GAO-01-633 Space Station
Appendix I: Comments From the National Aeronautics and Space Administration
Appendix I: Comments From the National Aeronautics and Space Administration
Page 16 GAO-01-633 Space Station
Appendix II: Staff Acknowledgments Page 17 GAO-01-633 Space Station
Jerry Herley, John Gilchrist, James Beard, Fred Felder, Belinda LaValle, Vijay Barnabas, Diane Berry, Rick Eiserman, Susan Woodward, and Cristina Chaplain made key contributions to this report. Appendix II: Staff Acknowledgments
// end //