The full report of an aviation safety review will be made available to “any interested party,” assured Michael Griffin, head of the National Aeronautics and Space Administration (NASA), following news accounts that information gleaned from surveys of 24,000 pilots was being suppressed out of concern that they could be misconstrued by a public skittish about the safety of flying.

A Freedom of Information Act (FOIA) request by the Associated Press for the data had been denied by NASA amid reports that its survey of pilots indicated roughly twice the rate of runway incursions reported by the Federal Aviation Administration (FAA), that the rate of bird strikes was far higher than officially recognized, that crew fatigue was chronic, and that general under reporting of aviation safety-related events was occurring (see Aviation Safety & Security Digest for previous account of this denial and the associated uproar). The pilot surveys were part of NASA’s National Aviation Operations Monitoring Service (NAOMS).

The public controversy prompted the House Committee on Science and Technology to hold a hearing 31 October, during which NASA Administrator Griffin backed off of the earlier denial of the data and promised full disclosure by 31 December 2007. Good thing, too, as Congress was clearly upset. Consider these not-very-veiled remarks of Rep. Mark Udall (D – Colo.), chairman of the Subcommittee on Space and Aeronautics.

“I am disappointed that we have had to convene today’s hearing. … (In) its response to the Associated Press’s requested for release of the NAOMS aviation safety survey data, NASA stated, ‘Release of the requested data … could materially affect the public confidence in … the air carriers and general aviation companies whose pilots participated in the survey.’

“That response is unacceptable. It’s certainly not in accordance with the National Aeronautics and Space Act of 1958, which created NASA and established objectives for the agency – one of which is ‘the improvement of the usefulness, performance, speed, safety, and efficiency of aeronautical and space vehicles,’ while directing NASA to operate in a manner that will ‘provide for the widest practicable and appropriate dissemination of information concerning its activities and the results thereof.’

“The NASA Administrator has since distanced himself from the language in NASA’s response to the FOIA request …I’d like to hear the Administrator reiterate that stance at today’s hearing.…

“As Chairman of the Space and Aeronautics subcommittee, I have oversight for both NASA’s aeronautics and aviation R&D programs and FAA’s aviation R&D programs.”

In other words, his subcommittee has control of the R&D budgets, and further mishandling of the NAOMS data could possibly result in fiscal penalties.

On 29 October Griffin had sent a letter to the full Committee declaring, “NASA believes that the data contains both confidential commercial data and information that could compromise anonymity that should be redacted prior to public release.” However, the Committee staff was unable to find a NASA or Battelle staffer (the contractor on the project) who could articulate what commercially sensitive or compromised anonymity resides in the data.

The bulk of the pilots responding to the survey were flying for the airlines, but about 4,000 general aviation (GA) pilots were included in a data-collection effort that ended in 2003. A survey of air traffic controllers was to have been undertaken, but it was never implemented. After spending more than $8 million to develop this tool and begin to put it in place, NASA shut it down before it went operational.

A reason for the premature demise was a lack of interest at the FAA. One reason for this may be the discouraging results of the pilot survey, which at the very least casts doubt on the efficacy of the FAA’s various safety oversight activities. However, this observation must be viewed in context. Not all the findings were bad news, and in some areas progress was being made; whether it was the result of FAA programs was another matter.

An FAA official was not present to testify at the hearing, although the FAA would be one of the prime beneficiaries of the NAOMS effort.

Capt. Terry McVenes of the Air Line Pilots Association (ALPA) did testify, arguing that release of the data would have been premature: “As in any first test, the data didn’t correlate very well with data from other sources, possibly due to the mix of general aviation and airline sources.”

This point of view was contradicted by those involved in designing the survey, who asserted that it was carefully crafted and provides useful insights. Moreover, airline and GA pilots were clearly separated and there is no melding of their respective inputs.

As far as Administrator Griffin’s caveat that the results have not received a peer review, sources close to the methodology say this is a smokescreen, and that peer review has been accomplished “many times” during the course of the NAOMS project.

More to the point, the initial suppression of the survey represents a “grave and dangerous challenge” to the issue of transparency when it comes to safety, testified Jim Hall, former chairman of the National Transportation Safety Board (NTSB).

What can be said is that the reports anonymously provided by pilots under the Aviation Safety Action Program (ASRP) are the grist for numerous safety alerts and corrective action throughout the year. The more structured NASA survey should provide a measurement of event trends over time. Because ASRS is based on voluntary reporting, much goes unreported and, consequently, ASRS cannot serve as the basis for trend analysis. The NAOMS data has the promise of doing that.

Below, extracts of the testimony of principal witnesses at the hearing:

Michael Griffin, NASA Administrator:

“Under federal law, NASA is required to protect confidential commercial information that is voluntarily provided to the agency and would not normally be released to the public. In preparing the response to the Associated Press’ Freedom of Information Act [FOIA] appeal, the characterization of the requested data by [NASA] Ames researchers raised concerns that the data likely contained confidential commercial information. This characterization was the basis for withholding the information under Exemption 4 [to FOIA].

“Considerable attention has been focused on one sentence in the final determination letter suggesting that data was being withheld because it could ‘affect the public confidence in, and the commercial welfare of, the air carriers and general aviation companies.’ I have already made clear that I do not agree with the way it was written. I regret any impression that NASA was in any way trying to put commercial interests ahead of public safety. That was not and never will be the case.

“I have directed that all NAOMS data that does not contain confidential commercial information, or information that could compromise the anonymity of individual pilots, be released as soon as possible. The release of this data will be accompanied with the proviso that neither the methodology nor the results have received the level of peer review required of a NASA research project. Therefore, the survey methodology and the data should not be considered to have been verified.

“NASA will receive a final report from Battelle by December 31, 2007 that will include a comprehensive description of the methodology, including approach, field trials, etc. NASA will make this report available to any interested party.”

Jim Hall, former chairman, NTSB. Hall was a member of the 1996 White House Commission on Aviation Safety & Security, which was headed by then-Vice President Al Gore:

“As a result of the Commission’s recommendation, NASA launched its $500 million Aviation Safety Program (AvSP), a partnership with the Department of Defense (DoD), FAA and the aviation industry, to focus on accident prevention, accident mitigation, and aviation system monitoring and modeling. It is this last point, the extremely important aviation safety research function, which brings us here today. Given a rapidly changing environment and a new set of risks, the attempt on the part of NASA to suppress safety data is a grave and dangerous challenge to the safety culture that has developed over the last century of aviation history, due to lessons learned from past aviation accidents and incidents.…

“As justification of its denial of a Freedom of Information Act request, NASA cited the potentially harmful affects on the commercial welfare of the air carriers and general aviation companies.

“Such an action runs exactly counter to the safety culture mentality the government and industry have worked to create over the past ten years … transparency forms the fundamental basis for any safety program. If we don’t know something is broken, we cannot fix it. If we do not know that runway incursions are actually occurring at a much higher level, then we cannot take steps and assign the resources to deal with them.…

“The culture of aviation safety has been built on constant critical self examination, in an open environment, with full sharing of all facts and analysis. Because we are safer today than yesterday does not mean that we cannot be safer tomorrow. It also doesn’t mean that our gains are not perishable. For example, on July 2, 1994 USAir flight 1016 crashed in Charlotte, North Carolina. We determined that the causal factor was something we hadn’t seen in the United States in almost a decade: windshear.

“Windshear detection equipment and improved pilot training had all but eliminated this hazard and yet more sophisticated weather detection equipment – Terminal Doppler Radar – had fallen years behind schedule due to procurement and design problems. …

“We must stay ahead of events instead of waiting for another crash. Steps must be taken to prevent a deterioration of our nation’s aviation safety culture, a deterioration that NASA’s denial of transparency plainly represents.”

“I was disappointed and perplexed when I learned that NASA decided the data collected by the NAOMS survey would not be released to the public. While I know that the most notable denial was that issued to the Associated Press, the Johns Hopkins University Center for Injury Research and Policy, a reputable safety research organizations in addition to being a leading scholarly institution, was also denied.…

“I also don’t believe the argument that NAOMS data are somehow limited or of no value because they are derived from a survey has merit. All data used for analysis, no matter its origin, have limitations and errors. Based on my experience, most if not all the databases used by the FAA for safety oversight and analysis contain errors and have limitation. This is why knowledgeable scientists and expert are involved in turning these data into useful information for decision makers.

“NAOMS data are no different in this regard. The NAOMS team made an extraordinary effort to clean and validate the data collected through the survey. The resulting data is of good quality and ready for meaningful analysis. Why would anyone decide that additional information, especially when it deals with the safety of the traveling public, should be hidden?

“Finally, the belief that the NAOMS data are not needed because the current safety oversight systems are adequate is untrue. Not all airlines have Flight Operational Quality Assessment (FOQA) programs or participate in the Aviation Safety Action Program (ASAP), a pilot based voluntary reporting system. Further, current safety oversight systems do not do a good job of measuring safety errors in the general aviation fleet, among small commercial operators, or among maintenance technicians, all of which have a direct influence on airline safety. A program like NAOMS can provide a unique oversight capability for all of the aviation system.…

“I believe NAOMS should be reinstituted and operated by an independent and unbiased organization. Such a program should receive funding directly from Congress to ensure its budget remains adequate to fulfill its mission.”

Jon Krosnick, professor, Stanford University, an expert in survey research and questionnaire design who was a member of the team that developed NAOMS:

“The instigation of NAOMS was a commitment made in the 1990s by the federal government to reduce the risk of commercial airplane crashes by a specific targeted amount in ten years. Once that target was set, federal agencies looked for ways to assess whether that goal would be achieved and realized hey had none. Simply tracking plane crashes would not be sufficient, because they happen extremely rarely and therefore do not indicated the amount of underlying risk posed by the many small events that, when cumulated, can increase the risk of an accident. ….

“Therefore, a new system for collecting information on the frequency of precursors to accidents was needed.

“NAOMS was designed to serve this purpose and to collect the needed information via high quality scientific and reliable surveys … who were watching the operation of the aviation system first-hand and who knew what was happening in the field.…

“As originally conceived by Battelle project manager Loren Rosenthal, NAOMS was to be a multifaceted survey project building on the Aviation Safety Reporting System (ASRS). For many years, ASRS has been a successful system for collecting anecdotal information from pilots about some of the risk-elevating events they witnessed. Each time an event occurs, a pilot can choose to fill out a form describing it briefly and mail the form to NASA’s ASRS office in Mountain View, California. An aviation expert then telephones the reporter to conduct a telephone interview to gather detailed information about the event.…

“Because pilots voluntarily choose to file reports on events, their choices about when to report and what to report are uncontrolled. And as a result, it is impossible to use ASRS to track trends in event rates over time. Therefore, NAOMS was envisioned to complement ASRS by producing accurate measurements of rates and trends…

“Every week of every year, NAOMS was planned to collect information from a representative sample of pilots flying commercial aircraft. The pilots would be asked to report the number of each of a series of different specific events that they had witnessed during a recent time period (e.g., the last 60 days). These counts could then be used to calculate the rates at which the events had occurred during that period throughout the entire air travel system.

“NAOMS had the potential to succeed especially because ASRS had already been successful. … But NAOMS was envisioned to go well beyond ASRS…

“The methods we used to develop the NAOMS questionnaire were state of the art. Indeed, the preliminary studies we conducted constitute valuable contributions to the scholarly literature on optimal survey design …

“NAOMS was intended to fill a hole by creating an ongoing pipeline of valuable information for the public and for the private sector to enhance the welfare of all Americans … Thank you for taking this opportunity to consider assuring that to happen.”