The airlines are complying with airworthiness directives (ADs) 98% of the time, according to the Federal Aviation Administration (FAA). In other words, all is well. But before accepting this declaration, consider a few factors, not the least of which is that the FAA conducted an audit of itself that lacks the test of independence.

One might say the FAA examined itself and graded the result an “A.”

In other words, a bit of healthy skepticism is in order when FAA Administrator Robert Sturgell maintained, “This audit gives us confidence that, overall, the system is safe and in almost every instance the airlines are complying with our safety directives.”

For one thing, and by no means the major thing, Sturgell made this declaration on a Friday afternoon, the perfect time to bury a story, as most media outlets have reduced staff on the weekends and newspapers and television news organizations have much of their weekend coverage already slated. Not surprisingly, Administrator Sturgell’s remarks appeared in a few wire service stories without benefit of critical analysis.

Sturgelll’s Friday, 5 September, press conference was late. The FAA was supposed to complete an audit of a small sample, about 10 ADs per carrier first, followed by an examination of airline compliance with 10% of ADs by the end of June. Allow 30 days for preparation of a report, and one would think the audit would have been completed and the results briefed to the airlines, to senior FAA leadership, and to the next senior agency, the Department of Transportation (DOT), by the end of July at the latest.

Remember that this review was undertaken in the wake of revelations of poor to nonexistent AD compliance at Southwest Airlines and American Airlines, whistleblower allegations of “too cozy” a relationship between the FAA and the airlines, Congressional hearings where this all came out to the huge embarrassment of the FAA, and legislation to establish an independent office within the FAA for allegations of collusion and nonfeasance to be investigated. (See Aviation Safety & Security Digest, ‘The FAA’s Failure to Protect the SKies,’ and ‘Agency Accuses Airline of Sloppy Maintenance While Sidestepping Its Own Safety Shortcomings,’ and ‘Agency’s Oversight of Airline Safety Under Scrutiny,’ and ‘Committee Vows to Legislate Changes to Strengthen Oversight of Airlines,’ and ‘Incestuous Relationships,’ archives.)

Since the results of this internal review were made public only in the most glowing terms, more than a month after it should have been completed and internally processed, it sure appears as if the results have been massaged. Sure enough, when the FAA was asked to provide a copy of the audit, the request was refused. An FAA official said the actual review contained proprietary information on the airlines, and as such, the review is not releasable. One wonders about making safety information “proprietary.” Thus, we have no idea which airlines were reviewed, which were not, what ADs were in the 10% sample, what other information was reviewed to assure the airlines were “safe,” and what strengths and weaknesses were found to the Air Transport Oversight System (ATOS), which is one of the principal tools for assuring airline safety.

Moreover, the 98% compliance rate was based on 88 instances of ADs not being properly followed at 11 airlines, but the 88 was later reduced to 24 ADs remaining under investigation after it was found that some carriers complied with the directives by other means or some of the planes had been put out of service. By this measure, the overall compliance rate was on the order of 99%+. Nor does this phenomenally high compliance relate to all ADs, just the small sample of ADs selected by the FAA. In any event, the FAA has not presented any evidence publicly to back up its very high compliance rate.

The documentation from the FAA is limited to two general press releases.

The next higher agency, the DOT, released a 69-page report and a press release calling on the FAA to implement 13 safety recommendations contained in the report. DOT Secretary Mary Peters hailed the Independent Review Team (IRT) as, just that, an independent assessment, whose recommendations, implemented between December 2008 and the end of 2010, should make the aviation system even safer than it already is.

“Today [10 September], the Independent Review Team has delivered a blueprint that will assure continued safe skies for America,” Peters proclaimed.

A close reading of the 13 IRT recommendations reveals a less comforting conclusion, that the initiatives are not as sweeping as called for by Congress.

For example, consider the very first recommendations:

“The FAA should retain the right to ground any plane not in compliance with an applicable AD. Inspectors should not be required or expected to conduct any type of risk-assessment before taking action on AD non-compliance.”

This recommendation rather belabors the obvious, as in “retain the right” to ground airplanes. One is not aware of this right being in jeopardy. The IRT might have added more punch by suggesting the FAA should ground an airline. Exactly this threat was made after the fatal crash in January 2000 of an Alaska Airlines MD-80, when it was discovered that the jackscrew controlling the horizontal stabilizer had not been lubricated. The FAA found egregious lapses in the carrier’s maintenance program and threatened to ground the airline unless meaningful reforms were made in maintenance management. In its June 2000 “show cause” letter to the airline, the cited the need for the carrier to revitalize and overhaul its Continuing Analysis and Surveillance System (CASS).

At the time, the FAA also vowed that it was going to evaluate CASS programs at other carriers.

Yet CASS wasn’t even mentioned in the IRT report.

Nor did the IRT mention the Service Difficulty Report (SDR) system, which would theoretically provide the FAA with insight into developing problems. For example, about 80 such SDR reports on the Eclipse EA500 Very Light Jet indicated numerous problems with avionics reliability. At a recent Congressional hearing, it was revealed that the EA500 jet design was certified with numerous extant IOUs, or deficiencies for which a promissory note was granted to fix them after the airplane entered service. Some of these deficiencies later surfaced in SDR reports. Yet the IRT did not mention the SDR system as a valuable tool to assess safety and maintenance. It should be mentioned that the SDR reports are a potential source of information on the safety implications of non-compliance with ADs.

But there are numerous shortcomings to SDR reporting. First, the SDR system is incomplete. Some airlines promptly file reports, as they are required to do. Other airlines file occasional reports; some carriers do not file at all.

Further, if the SDR reports are passed through another filter, that of the FAA’s requirement to record an “apparent cause,” then the requirement both to file and to record the apparent cause is cut in half. The SDR system as a tool for hazard analysis is severely compromised by its very reporting protocol.

Moreover, airlines are only required to submit SDRs on malfunctions that occur in the air. An FAA proposal to submit SDRs for malfunctions that occur on the ground met with fierce industry resistance and was never implemented.

As a consequence of these shortcomings, the SDR database is not a useful tool for FAA analysis of industry-wide incidents and trends. Thus, while Nicholas Sabatini, associate FAA administrator for safety, told Congress recently that it would be “irresponsible and reckless” to wait for an accident, one might argue that it is equally “irresponsible and reckless” to tolerate the gaps in SDR reporting that would give the FAA insight into accident precursors.

The limited usefulness of the SDR has prompted the National Transportation Safety Board (NTSB) to complain about the lack of comprehensive and timely data to the FAA numerous times. The NTSB’s pleading have had no effect on improving the effectiveness of the SDR database. In June, the FAA took the database offline with the claim that it was “hacked.” That assertion needs to be assessed by the fact that since 4 June the SDR database has been unavailable to the worldwide aviation community.

With SDR inoperative, about all that’s left to assess the safety of carrier’s operation is the Air Transport Oversight System (ATOS). The IRT reviewed ATOS and concluded that 70%-80% of an inspector’s time is spent at the computer terminal simply entering data, leaving too little time for analysis of hazards or safety trends. The IRT recommended that the FAA “study” ATOS. The FAA has agreed to commence such a study in March 2009.

Not only is this delay inexplicable – one would wish the “time/motion” study began immediately – both the IRT recommendation and the FAA response are totally inadequate to the real problems with ATOS. These are documented on page 53 of the IRT report:

“(It) seems that ATOS is not yet supporting the risk-management process in the manner originally envisaged. Many inspectors reported to us that they do not trust the quality of data within ATOS, in part because the only way to get through the data-entry task in a reasonable amount of time is to answer ‘yes’ to most of the questions, so one can move on. (Many of the questions within the certification module have the form ‘Does the carrier operate an adequate system for ensuring xxx?’ If an inspector answers ‘no,’ they [sic] are obliged to provide an accompanying test-based description of the inadequacy.) Also, inspectors report that the system’s logic and data-structure is inflexible, with the consequence that important observations about hazards identified – if those hazards do not relate specifically to one of the certification conditions – do not fit easily within the logical and hierarchical structure of ATOS, and therefore require extra effort to support.

“ATOS does have a separate Risk-Management Module, in which an inspector can report the discovery of a particular hazard. The data fields within this module are free-text format, so that any risk or hazard, of any shape or size, can in theory be reported. In practice, the Risk-Management Module is not heavily used, as the scheduled and routine parts of ATOS-directed inspection activities take up most or all of the time available. Moreover, given the free-text format, the data contents of the Risk-Management Module cannot easily be aggregated or analyzed to reveal emergent trends or patterns across the industry. By contrast, ‘yes’ and ‘no’ answers to standardized questions are easy to count, and to aggregate, and to compare across carriers, regardless of how accurate or meaningful these answers might be.

“Based on our brief examination of the Risk-Management Module, it is not clear to the IRT whether entering a hazard into this module prompts the type of rigorous analysis, open-minded search for mitigation strategies, and monitoring of impact that a mature risk-management approach would require. The fact that this module exists does not guarantee its effective use …

“We believe ATOS requires substantial further improvement …”

It should be noted that this indictment is levied on a system the FAA has had in place since 1998.

The IRT was also silent on the matter of the National Aeronautics & Space Administration (NASA) effort to compile safety-relevant trends. This pioneering project was shut down for lack of FAA support. (See Aviation Safety & Security Digest, ‘Chance for Valuable Insights Into Safety Issues Lost,’ archive, May 2008.)

If Secretary Peters expects anything substantive to come from the IRT, she is sadly mistaken. The deadlines are so far into the future that accountability for implementing them will be lost completely.

Peters would have been far more effective simply demanding that the FAA implement two ideas from her own Inspector General (IG): to periodically rotate inspectors to impede the development of comfortable relationships between FAA inspectors and the airlines they oversee, and to establish an independent office within the FAA for inspectors in the field to report safety problems free from the negative influence that now prevails. Whistleblower allegations are presently handled by the same offices at FAA headquarters that muzzle the reports and take retribution on the persons submitting the reports

The Safety Issues Reporting System, or SIRS, operational since April, has the FAA principal inspector and the certificate management office receiving the SIRS reports and evaluating them. In other words, higher offices in the same chain of command will determine whether SIRS reports have any merit.

On top of that, Peters could have waited until the IG has completed its own review of FAA oversight, ongoing since 2 June and not yet complete, before pronouncing the oversight of aviation maintenance and AD compliance fully satisfactory, given just a few tweaks recommended by the IRT.

The IRT did not recommend what Congress wants. In July the House unanimously passed the Aviation Safety Enhancement Act of 2008 that establishes an independent Whistleblower Investigation Office within the FAA. This office is to “receive complaints and information submitted by [FAA] employees …. relating to a violation of an order, regulation, or standard … [regarding] aviation safety.” The office is empowered to investigate and report directly to the FAA administrator and/or the DOT/IG. There is a specific passage in the legislation regarding the independence of this new office:

“The Secretary, the Administrator, or any officer or employee of the Agency may not prevent or prohibit the Director from initiating, carrying out, or completing any assessment of a complaint or information submitted …  or from reporting to Congress on any such assessment.”

As Gabe Bruno, a retired FAA manager and now independent safety consultant observes, “The IRT report falls short of demanding accountability from FAA leadership for the poor state of affairs at the agency.” The fundamental problem, one could posit, is one of professional ethics. When one sees regional FAA officials testifying to Congress that their leaders in Washington have shortstopped, obfuscated, or denied any safety problem, the problem transcends technical shortcomings in processes like ATOS. (Bruno’s detailed observations are at ‘Opening the Book on Leadership Dysfunction’)

Note that the word “accountability” was not used by Secretary Peters or Acting Administrator Sturgell. Yet that is precisely what Congress seeks by mandating the creation of the Aviation Safety Whistleblower Investigation Office. The bureaucracy is studiously avoiding the one thing Congress most desires: some high-level accountability for safety.

(The “independent” audit, Managing Risks in Civil Aviation, may be viewed at http://www.dot.gov/affairs/IRT_Report.pdf)