Letter from IFPTE To Rep. Gordon Regarding NASA Aviation Safety Research Projects

November 29, 2007

Hon. Bart Gordon, Chairman
House Science & Technology Committee
2320 Rayburn House Office Building
Washington, DC 20515

Dear Chairman Gordon,

The International Federation of Professional and Technical Engineers (IFPTE) is the largest Federal Employee Union at the National Aeronautics and Space Administration (NASA) and has the honor and privilege of representing thousands of scientists, engineers, and technical employees across the Agency. Our NASA locals are deeply committed to maintaining technical excellence, independence, and integrity at NASA's field centers. We therefore thank you for this opportunity to submit for the record our response to the NASA testimony at your October 31st 2007 hearing on Aviation Safety.

At the hearing, the Administrator testified that a NASA Aviation Safety research project had serious technical flaws and publicly questioned its validity. IFPTE takes any criticism of NASA science very seriously. As a result, we used our confidential access to key employees to perform our own investigation of the facts surrounding this matter. We are now pleased to report that our findings on this issue differ from what was stated at your hearing. Regardless of the circumstances, we at IFPTE feel that it was inappropriate for him to disparage NASA research in public, especially at a hearing triggered by mistakes made by NASA HQ's senior management and legal teams, not by any action of the project's technical staff.

Our primary concern is that the American taxpayer will be deprived of the fruits of millions of dollars and years of valuable Aviation Safety research and development (R&D) because of repeated judgment failures by NASA's senior leadership. There was no valid legal basis for the blanket refusal to honor a legitimate FOIA request for public safety data. There was no valid programmatic basis for the decision either to terminate the project before the R&D team completed its analysis of the pilot survey data or to transfer NASA's promising safety monitoring technologies to entities ill- suited to capitalize on their full potential. We can now add to this list that there was no valid scientific basis for the Administrator's technical criticism of the National Aviation Operations Monitoring Service (NAOMS) project. Given this track record of poor judgment and biased decision making, we at IFPTE welcome Congressional intervention and hope that aggressive oversight may bring about an urgently needed change in course for NASA's Aeronautics Research Mission Directorate (ARMD).

Our goal with this letter is to summarize the key events as we see them, to provide a point-by-point rebuttal to the assertions made by the Administrator, and to expand the discussion to address potential root causes ranging from flawed ARMD policies to FAA interference, and finally to suggest a path forward to remedy the situation.

A continuing saga of missteps

It does not take a rocket scientist to understand that categorically refusing to release government-financed aviation safety research data to the public using the argument that it might adversely affect corporate profits is egregiously wrong. We are pleased that all the raw data have been released to your Committee and that NASA has pledged to release all of the properly de-identified data to the public by December 31st, 2007. It is regrettable that this was not NASA HQ's initial decision.

It is, however, worrisome that NASA now appears to be hedging on its public promise of full release of all legally releasable information before the end of the calendar year. It is ridiculous to argue that NASA will take a year and needs to hire expensive outside consultants to handle a data-filtering process that can best be handled by those aviation experts who designed and implemented the survey and who are most experienced at releasing de-identified aviation safety reports. Taxpayers are much more likely to trust this process if it is performed by scientists in the trenches motivated by their passion for aviation safety than if it is led by political appointees or senior managers who cannot be expected to render an opinion that conflicts with that of the Administration.

It is unfortunate that the hearing did not focus on NASA HQ's FOIA refusal, but rather was sidetracked by a public repudiation of the NAOMS project by the Administrator himself. It is troubling that these unfounded attacks coincided with NASA HQ finally agreeing to release the information. The Administrator provided only a few selected anecdotal data points that were misrepresented to support his argument. One can only infer that this misdirection was to pre-spin anticipated queries about the Administration's policy decision to terminate NAOMS (and other promising Aviation Safety projects). Most importantly, they did so without assuring their proper transfer from R&D to operational activities run by entities with the appropriate independence, resources, and credibility to acquire and analyze such large and complex databases.

IFPTE's position has been that, regardless of NASA HQ's improper FOIA refusal, the press was inflaming public opinion with an unscientific reading of a selectively leaked subset of data. In an October 29th letter to you and Chairmen Udall and Miller, IFPTE pointed out that it would be inappropriate for anyone to comment on the data until they could be properly analyzed. In particular, as Dr. Krosnick stated in his testimony, the data must be properly statistically corrected before any extrapolation to overall rates. The press might be excused for using hyperbole to engage the public's interest in topics of significant impact to society and public safety. However, the use of exaggeration and selective attention in NASA's testimony was inexcusable.

NASA HQ's now-public bias against NAOMS raises serious concerns about what role that bias might have played in the decision to terminate the project and subsequently to refuse to release the data. Was the final FOIA refusal letter reviewed by the Deputy Administrator or other top managers before being sent? It would now appear that the Administration's FOIA refusal was clearly consonant with a then-secret fear that the NAOMS data might be used to challenge the status quo of "accepted" FAA databases. Indeed, the NAOMS data have the potential to expose the underestimation of safety precursors within FAA databases. Thus, the extensive NAOMS dataset should be analyzed by aviation research professionals, independent of the FAA, of the airlines, and of meddling by political appointees, so that the public may derive the best-possible independent and unbiased scientific estimation not only of the safety risks to our National Airspace, but also of the accuracy of existing FAA databases.

Although, prior to the October 31st hearing, IFPTE found no compelling evidence that Aeronautics program or line managers took action specifically aimed at suppressing NAOMS data and the culpability appeared confined to NASA's legal team, the Administrator's shocking testimony and our subsequent findings now give us serious pause. Rather than explaining how the FOIA legal missteps occurred and what steps were taken to prevent any future repeat of this fiasco, the Administrator dodged the question of how to correct a broken FOIA-response process and instead began to unfairly tarnish NAOMS, its local center management, its data, and its methods. Fortunately, your Panel II witnesses, all much more familiar with NAOMS and its methods, provided a clear and systematic refutation of NASA's testimony. The Administrator's allegations can be dismissed once the facts are made clear.

Clarifying the Administrator's testimony

• The termination of NAOMS was a change in policy. Although the former Aviation Safety and Security program funding was only allocated according to a standard 5-year plan, the NAOMS project was planned as an ongoing activity with clear approved milestones slated to expand beyond the pilot survey to cover a wide range of aviation workers. Although funding past fiscal year 2004 (FY04) was not initially specified, this is not an indication that NAOMS activities were slated to end and indeed the expectation was that it would mature into an ongoing operational system, albeit the follow-on process was not fully defined at that time. When a Federal Aviation Administration (FAA) team met with NASA's NAOMS team in August of 2003, the lead FAA representative's recommendation was that "the FAA support the NAOMS through active participation on the NAOMS Working Group [at least initially]...It is estimated that the group will conduct three meetings during the first year, followed by two meetings per year, thereafter." It is hard to reconcile that statement with the assertion that NAOMS was scheduled to be terminated in September of 2004 after completion of the pilot survey. NASA HQ's argument is meant to conceal its policy decision to terminate NAOMS by transferring this activity to the AirLine Pilots Association (ALPA), which clearly does not have the financial or technical resources either to continue the survey, or to expand it to the full range of stakeholders, or to perform the necessary scientific analysis of the data. The plan was to transfer NAOMS (along with other Operational Safety projects) so that NASA could testify that it had not killed NAOMS.
• The scope of NAOMS was not limited to the development of methodologies. The statement made by the Administrator that NAOMS had an "overarching goal of developing methods" misrepresents the full scope of NAOMS, yet this misleading talking point was disseminated to the press even prior to the hearing: "A NASA spokesman, David Mould, said Monday that the purpose of the study was 'to develop methodologies' for analyzing data, not to find what the data would show. [New York Times,10/23/07,query.nytimes.com/gst/fullpage.html?res=940DE3DC1F38F930A15 753C1A9619C8B63]" It is simply not credible that NASA would undertake a survey with tens of thousands of interviews costing millions of dollars just to study proven survey methods and to ignore what the data would show. The purpose of NAOMS was to develop enabling technologies and to collect and analyze data in order to demonstrate an ability to identify safety precursors not available in existing databases. The development and validation of the specific methods were necessary first steps in that effort, but were not the ultimate goal. Above and beyond the tailoring of proven survey methods to an aviation setting, NAOMS was specifically tasked to "demonstrate the value of the pilot survey", a task that requires that the data be analyzed and assessed by subject matter experts in order to determine how reliably the survey identified precursor "pointers" to safety problems. However, the Administration decided to terminate the project and directed that the transfer of the project to ALPA become the primary focus, which is a major reason why the analysis and reporting of the results (now secondary) were delayed. Since the hearing, NASA HQ has directed that the NAOMS contractor deliver a "final" report, with no NASA affiliation, narrowly focused on describing the survey methods. This direction blocks NASA's Operational Aviation Safety experts from completing the planned analysis of the data and effectively prevents the release of any official NASA report of analyzed NAOMS data. How does the taxpayer benefit from preventing this planned analysis of NAOMS data?
• The main objective of NAOMS was to monitor trends. Independent of any extrapolation to overall rates, the NAOMS data were designed to monitor trends in safety precursors as changes are made to the National Airspace System (e.g., new cockpit technologies or new air traffic management policies introduced). As the nation moves forward towards the radical changes envisioned within NextGen, it is absolutely essential that we have an ongoing sensitive and statistically valid tool to monitor these trends so that we can identify problems occurring at the human-system level before they become accidents. That is the main and urgent rationale for the ambitious full-scale NAOMS surveys. No existing aviation safety data monitoring system has access to the experiences of the full range of key players in the system (e.g., pilots, air traffic controllers, maintenance workers, etc...) needed to properly assess trends in safety precursors across the entire aviation system in a statistically sound manner. NASA HQ's decision to terminate NAOMS is a lost opportunity to have a more complete picture of the health of the overall aviation system to help proactively mitigate risks to the flying public, especially as we move forward to modernize the National Airspace System.
• The data from NAOMS could also be used to estimate overall rates. Although the analysis and extrapolation of raw NAOMS data to estimate overall rates has not yet been completed by the NAOMS team and there remains honest disagreement between experts on how and when valid rate estimations can be made, such extrapolation is possible in certain cases under proper circumstances including statistical correction for possible multiple observations (see Dr. Krosnick's testimony). Although NAOMS-generated overall rates would not be fully devoid of error even when properly statistically treated, they would represent a powerful independent dataset that could be compared with equally imperfect FAA rates. It is through the convergence of multiple independent approaches that we can have a more accurate picture of the true functioning of the system. In his effort to discredit NAOMS, Dr. Griffin mischaracterized two examples of rate estimation from the NAOMS data: engine failures and unruly passengers. The issue of the improper statistical treatment of the leaked data aside, he made inappropriate apples-to-oranges comparisons between NAOMS and other data across substantively different questions. The NAOMS data represented any kind of engine shutdown (even if followed by a re-start) for a wide range of aircrafts and pilots; a comparison was made to the rate of complete engine failures on commercial jets that are obvious public events. The NAOMS data counted impacts on flight planning from unruly passengers including expedited arrivals; a comparison was made to the rate of emergency diversions and landings that make the evening news a few times a year. The FAA has had misgivings from the start regarding any calculation of rates because the independent cross-checking potentially possible with NAOMS data would threaten the FAA's monopoly on the rates of safety-relevant "events". It is, however, absolutely essential that the FAA, a regulatory organization, and the airlines, for-profit organizations, not be the sole collectors of aviation safety data because confessions and reports of problems to such policing organizations are necessarily underreported despite the FAA's untenable position that it counts all such "events". The FAA's assertion that it keeps perfect data is being challenged by recurring media reports of underreporting of supposedly objective "events" in FAA databases [e.g., USA Today 6/23/05, www.usatoday.com/news/washington/2005-06- 23-air-safety-violations_x.htm]. Furthermore, even peer-reviewed academic studies are finding "event" rates inconsistent with FAA numbers. For example, a recent study of runway incursions observed rates about two orders of magnitude higher than those expected from the FAA 2007 runway safety report. [Jeddi et al., 2006, catsr.ite.gmu.edu/pubs/JeddiShortleSherryICRAT06.pdf ]
• The NAOMS survey methods have been extensively peer reviewed. Standard peer review is typically performed post hoc when a final manuscript is submitted for publication to a technical journal, so the criticism in NASA's testimony was simply inappropriate. Methods alone are not generally peer reviewed unless they are dramatically different from anything used previously. NAOMS did not invent new survey methods; it adapted proven survey methods to a new domain. These methods have been widely used in similar contexts in published scientific studies that have been extensively peer-reviewed (see Dr. Krosnick testimony and Curriculum Vitae). In addition, NAOMS was thoroughly reviewed and approved by the Office of Management and Budget (OMB). Lastly, NAOMS established a collaborative team with NASA, FAA, academic, and private sector aviation and survey experts that reviewed, tested, and validated the specific survey questions and methods prior to initiating the large scale pilot survey. NAOMS' methods have been more thoroughly reviewed by a wide array of internal and external experts than just about any other NASA project. After the pilot survey data are analyzed and a report is written up by the NASA NAOMS investigators would be the proper time for a formal peer-review.
• The use of non-experts as NAOMS interviewers was appropriate. The NAOMS survey questions were designed to be clear and unambiguous, and this was validated in trial studies. The interview process was simply to systematically walk through these highly polished questions. Professional interviewers without aviation safety knowledge provide unbiased and trustworthy interfaces, who do not attempt to "clarify" or "spin" or "color" questions or answers. The use of aviation experts, potentially biased by their own perspectives or experiences, could have compromised the integrity and uniformity of the data. The Aviation Safety Reporting System (ASRS) interviews are quite different in that the purpose of the "call-backs" is to clarify and expand upon statements made in the initial report. This explicitly requires an aviation expert to tease out additional information from ambiguously worded or incomplete narrative statements and technical jargon.
• The NAOMS data are not available in existing databases. NASA's independent collection of human-system safety precursor data was thoroughly reviewed and approved by the OMB, the executive Agency specifically tasked with assessing redundancy in government projects. Indeed, if fully implemented, the NAOMS database would be the most extensive and complete dataset documenting system safety precursors. Also, all safety databases have weaknesses and underlying assumptions or biases; it is through the convergence of multiple, interleaved, sets of information that one can derive the most accurate total picture of the health of the National Airspace System.
• The National Academy of Science (NAS) recommended continuation of NAOMS. The Administrator provided a selective reading of the NAS 2004 report on NASA's Aeronautics Technology programs to imply that the NAS did not support NAOMS. Mindful of tight budgets, concerned about the chronic underfunding of the ASRS, and eager to see efficient joint use of limited ASRS and NAOMS resources, the official NAS recommendation on NAOMS was that "NASA should combine NAOMS methodology and resources with the ASRS program data to identify aviation safety trends." [p. 100, NAS 2004 report]. Dr. Griffin's testimony omitted this overall endorsement of NAOMS as a powerful new way to monitoring safety trends, and instead selectively amplified the few NAS concerns.

The NAOMS crisis as pointer to systematic problems with ARMD planning

The FOIA-refusal fiasco and the misleading testimony aside, the NAOMS crisis represents the tip of an iceberg of adverse consequences generated by policy decisions within ARMD. Although we found no evidence that Associate Administrator Porter engaged in improper behavior and although she provided limited bridge funding into FY06 to foster the transfer of NAOMS to ALPA, this in no way relieves her of culpability for policy decisions that have systematically harmed NASA's core capabilities and competencies in Aviation Safety research and that have contributed to the orphaning of promising new critical technologies and data.

In Dr. Porter's testimony for the record to your Committee on September 26th 2006, she revealed the plan for continued dramatic reductions in the net Aeronautics programmatic spending authority from $788 million (actual FY04) to $451 million (proposed FY07). This near-halving of NASA's Aeronautics R&D spending power over three years has only been avoided thus far because of repeated Congressional rescues, yet Congress has not succeeded in preventing a radical "reorganization" of ARMD. In particular, the Aviation Safety program has become excessively aircraft-centric. Despite the fact that human-system interaction failures remain the predominant cause of aviation accidents [e.g., Statistical Summary of Commercial Jet Aircraft Accidents: Worldwide operations 1959-1995, Boeing, 1996], system-centric Operational Aviation Safety research, such as that performed under NAOMS, has been targeted for funding reductions, resulting in the loss of key personnel and capabilities [USA Today, 9/27/06, www.usatoday.com/tech/science/space/2006-09-27-nasa-nstb_x.htm]. For example, it is ironic that the Committee was so horrified by the ASRS report of a pilot and co-pilot falling asleep at the controls after a number of back-to-back red-eye flights, when the Fatigue Countermeasures research group at NASA's Ames Research Center has been disbanded for more than a year. Because of the ARMD budget cuts, NASA's capability in fatigue countermeasures R&D is no longer available to help prevent that frightening flight scenario. A similar fate has been visited upon Icing research group at NASA's Glenn Research Center, which, among other things, has been developing systems and training to enable pilots to recognize, avoid, and mitigate dangerous icing conditions, a serious safety concern for regional turboprops and General Aviation. Both Icing and Fatigue feature in the National Transportation Safety Board's (NTSB) Most Wanted Safety Improvements List [http://www.ntsb.gov/recs/mostwanted/aviation_issues.htm]. The loss of long-standing, well respected Agency capabilities such as these, at NASA's traditional Aeronautics Research Centers, is not good for the safety of the flying public and is a direct consequence of Dr. Porter's misguided policies.

FAA interference

It is quite revealing that the slandering of NAOMS had already begun prior to your October 31st hearing, largely promulgated by FAA Deputy Associate Administrator for Aviation Safety Gilligan, who stated that "her agency had been briefed on the survey results but had not examined the underlying data. But she cast doubt on its value..." [New York Times, 10/23/07, Ibid]. Having not examined the underlying data, why was she so eager to criticize the value of NAOMS in the media and why does Dr. Griffin now appear to be quoting from the same talking points? Why is the FAA so afraid of the cross-checking of FAA data by independent sources? NASA HQ's FOIA refusal and the Administration's coordinated attack on the NAOMS data gives credence to the concern that political interference with NAOMS may have been orchestrated from above. We were pleased to see that Chairman Lipinski began to connect the dots when he saw a potential link between the FAA and NAOMS' cancellation prior to performing the Air Traffic Controller portion of the survey. Could it be that FAA management, which was contemplating changes in work rules and staffing for Air Traffic Controllers, did not want a survey method in place that could track the impact of these changes on safety precursors? [see Washington Post, 11/17/07, www.washingtonpost.com/wp- dyn/content/article/2007/11/16/AR2007111601857.html]. Dr. Gilligan's public criticism of NAOMS is particularly inappropriate given that, from at least 2003 onward, FAA technical staff has supported NAOMS and was integrally involved in the preparation and planning of the NAOMS survey through joint meetings that included NASA, FAA, NTSB, academic, and private sector aviation and survey experts. Indeed, this active and sustained participation provides clear evidence that FAA technical personnel considered NAOMS to be a promising new tool to monitor the impact of new safety enhancements introduced into the Airspace System.

A path to recovery

IFPTE recommends that your Committee:

• Direct NASA to provide testimony for the record describing what steps it has taken to reform NASA's legal process for evaluating FOIA requests and to ensure that improper FOIA refusals not occur in the future;
• Direct NASA to provide immediately at least $150,000 of un-obligated carry-over fiscal year 2007 (FY07) funds to the NASA NAOMS investigators so they (in collaboration with their academic and private-sector partners) can complete a rigorous preliminary analysis of the data with a report to be provided directly to your Committee (and to the Commerce, Justice, and Science Appropriations Sub-Committee) without HQ interference, no later than September 30, 2008 (so as to end the speculation and to provide critical technical guidance for future independent studies of the database by clarifying the best practices regarding statistical handling of the data);
• Direct (in conjunction with the Transportation Committee) the Government Accountability Office to investigate what actions the FAA may have taken to undermine NASA's safety data collection and analysis efforts in general and in suppressing the NAOMS data in particular, as well as any cooperation between FAA and NASA senior management in the decision making on the handling and release of the NAOMS data;
• Urge the House and Senate leadership that the final Omnibus FY08 Spending bill include the $146 million Aeronautics budget increase over the President's budget within the House version of H.R. 3093; and
• Authorize NASA's FY09-11 budgets to fund the continuation of NAOMS data collection/analysis and direct that NASA adopt the NAS recommendation to combine NAOMS with ongoing ASRS activities (with a subset of the funds used to cover a NASA Research Announcement inviting both internal and external aviation safety researchers to apply for grants to perform peer-reviewed analyses of the NAOMS data). The final funding decision would of course be up to NASA's Appropriators who would be able to interpret the Authorizers' guidance with the added benefit of having the results of the NASA NAOMS report described above in hand.

NASA, leveraging its special relationship with academia and the stakeholder community, is best suited to lead the collection and analysis of system-wide aviation safety data, independent of the conflicting interests of the FAA's regulatory function. It is only because of NASA's hard-earned reputation for impartiality, technical excellence, respect for anonymity, and dedication to aviation safety that it can play the critical role of a trusted, honest broker working with very sensitive data voluntarily provided by key stakeholders (e.g., pilots, air traffic controllers, maintenance workers) who are vulnerable to retaliation from the FAA and/or the airlines. The NAOMS crisis has the potential to harm NASA's relationship with these stakeholders, carefully nurtured over decades. We hope that any steps taken by Congress and/or NASA HQ will be mindful of that important fact.

We thank the Committee for investigating this incident. Unfortunately, the hearing provided NASA with a very public podium to disseminate misinformation about NAOMS (that continues to reverberate through the media, unfairly undermining the public's confidence in NASA and its scientists) and to deflect attention from the Administration's culpability in refusing to release of aviation safety data. We fear that, unless Congress acts to correct the current course of events, the net outcome will be the discrediting of NAOMS' valuable data and the orphaning of its powerful methods, the potential destruction of the reputations and careers of dedicated public servants, the further marginalization of NASA's Aeronautics research and demoralizing of its scientists, and the emboldening of NASA HQ to further defund NASA's independent Operational Safety research.

I thank you for your consideration. Should you have any questions please feel free to contact me, or IFPTE Legislative Director Matt Biggs at (301) 565-9016.

Sincerely,

Gregory J. Junemann,
President

Cc: Hon. Ralph Hall
Hon. Mark Udall
Hon. Tom Feeney
Hon. Brad Miller
Hon. James Sensenbrenner
Hon. Dan Lipinski

Please follow SpaceRef on Twitter and Like us on Facebook.