From: NASA Office of Inspector General
Posted: Monday, March 31, 2008
The genesis of NAOMS was rooted in the February 12, 1997, White House Commission on Aviation Safety and Security's "Final Report to President Clinton." For the next several years, NASA collaborated with the Federal Aviation Administration (FAA) and the National Transportation Safety Board to examine options and make recommendations concerning potentially useful technologies. Technologies recommended for the identification of existing accident precursors in the aviation system and for forecasting potential safety issues, which NASA developed in parallel and interdependence, became the Aviation System Monitoring and Modeling (ASMM) Project.
The ASMM Project, established in 1999, consisted of four elements: Data Analysis Tools, Intramural Monitoring, Extramural Monitoring, and Modeling and Simulations. The Extramural Monitoring element of ASMM consisted of NAOMS and the Aviation Safety Reporting System (ASRS). Because ASRS is a voluntary reporting system and not suitable for comparative statistical analysis of changes in the safety of the National Aviation System, ASMM Project management designed NAOMS to complement and enhance the ASRS information. NAOMS would include survey data from pilots, mechanics, air traffic controllers, and others, which could provide insights into the performance and safety of the National Aviation System. By routinely evaluating the data, decision makers of the aviation community could quantitatively measure safety, assess trends, identify factors driving those trends, and evaluate the effects of new technologies and procedures introduced into the National Aviation System.
NASA's Ames Research Center contracted with Battelle Memorial Institute (Battelle) to develop NAOMS in fiscal year (FY) 1998 as an additional task under the established ASRS contract. Research and development for a NAOMS survey took place through FY 1999 and into FY 2000. In early 2000, Battelle concluded a field trial of the survey that included 630 responses from air carrier pilots. After Battelle incorporated suggestions and recommendations from aviation community stakeholders, it began the large-scale NAOMS research by implementing the air carrier pilot survey in April 2001. Battelle extended the survey to general aviation pilots the following year. The NAOMS survey was not implemented among any of the other intended survey groups.
Collection of the NAOMS survey data concluded in December 2004. Approximately 30,000 surveys had been completed, consisting of approximately 25,000 air carrier pilot interviews and 5,000 general aviation pilot interviews. Although Battelle and NAOMS Project management had done preliminary analysis of the data, working groups designed to validate the data did not accomplish this goal. As a result, the value of the NAOMS survey data as a potential contributor to the prevention of aviation mishaps has not been determined.
In 2004, NAOMS Project management began work to transition NAOMS to a Web-based application and prepare it for permanent service. The Joint Implementation Data Analysis Team (JIMDAT) of the Commercial Aviation Safety Team (CAST)3 showed particular interest in the NAOMS Project for monitoring and measuring safety enhancements introduced into the National Aviation System. Following the successful demonstration of the application, ALPA offered to operate NAOMS on behalf of CAST and accepted NAOMS as a Web-based application for permanent service in January 2007.
Following the October 2007 publication of the Associate Press article, the NASA Administrator appeared before the House Committee on Science and Technology on October 31, 2007. Questions posed by Congress centered on the survey confidentiality and the release of the data. The Administrator's testimony included a promise to release and publish the NAOMS data. On December 31, 2007, the redacted NAOMS survey data were posted on NASA's Web site. Although the Web site contains some previously unreleased information, the Web site does not adequately articulate the purpose of the NAOMS Project and its relationship and contribution to the larger ASMM Project.
We found that NAOMS Project management conducted the research, development, and implementation of NAOMS in accordance with NASA Policy Directive (NPD) 7120.4A, "Program/Project Management," November 14, 1996, and NASA Procedures and Guidelines (NPG) 7120.5A, "NASA Program and Project Management Processes and Requirements," April 3, 1998. However, we found several project management deficiencies that occurred over the life cycle of NAOMS. Specifically,
Additionally, we found that NASA had not adequately described the designed and intended uses of NAOMS data. Specifically, as of February 2008, NASA had not published an analysis of the NAOMS data nor adequately publicized the details of the NAOMS Project and its primary purpose as a contributor to the ASMM Project. Consequently, the NAOMS survey data, which were intended to be just one part of the ASMM Project, could potentially be taken out of context and misunderstood in relation to identifying aviation risk.
The Government may have missed an opportunity to foster a deeper understanding of the aviation safety environment from 2001 through 2004 because its working groups were unable to reach a consensus on the validity or value of the NAOMS data. As a result, NASA was reluctant to publish a report detailing research and conclusions garnered from the collected NAOMS survey data.
Our February 19, 2008, draft of this memorandum recommended that the Associate Administrator for the Aeronautics Research Mission Directorate (ARMD) take the lead concerning NAOMS data and determine whether the data were useful for NAOMS' intended purpose: to produce data from which trends might be identified and considered in concert with other ASMM activities. Additionally, we recommended that the Associate Administrator for ARMD ensure that NAOMS Project research is published and that he make NAOMS-related information available to all interested parties.
In commenting on the draft of this memorandum (see Enclosure 3), the Associate Administrator concurred with our recommendation to determine whether NAOMS accomplished its intended purpose of producing data from which aviation safety trends might be identified, as well as our recommendation to release and post on NASA's public Web site NAOMS-related information. We consider these two recommendations resolved.
The Associate Administrator nonconcurred with our recommendation to publish a detailed report analyzing the NAOMS research, to include findings and conclusions gained from the survey data, preferring to focus resources on better understanding the validity of the survey methodology. Management's planned action is partially responsive to the intent of our recommendation, which was to provide closure for the NAOMS Project. Accepting the Associate Administrator's view that understanding the validity of the NAOMS survey methodology may prove to be the most valuable component of the NAOMS Project, we revised the draft recommendation. Our revised recommendation makes the publication of a detailed report analyzing the NAOMS research contingent upon the results of ARMD's planned independent assessment of the NAOMS methodology. We request that management provide additional comments in response to this revised recommendation by April 30, 2008.
// end //