The inspector general of the Department of Transportation (DOT/IG) is delving into the Federal Aviation Administration’s (FAA) maintenance oversight system, which in many cases allows airlines to self-report problems.

FAA oversight of the airlines was severely criticized in Congressional hearings last April, where various mid-level FAA whistleblowers charged the agency with being too cozy with the airlines and was not acting with sufficient rigor in the face of safety deficiencies (see Aviation Safety & Security Digest, ‘Committee Vows to Legislate Changes to Strengthen Oversight of Airlines’ and ‘Incestuous Relationships’).

According to the 24 June memorandum by the DOT/IG, the “series of audits” to be conducted is a direct result of those hearings, “The House Committee on Transportation and Infrastructure and the Senate Subcommittee on Aviation Operations, Safety & Security requested that our office review the FAA programs and oversight weaknesses that contributed to the situation at Southwest Airlines.”

During the hearings on FAA oversight of maintenance at Southwest Airlines, witnesses testified to similar lapses at American Airlines. Thus, the DOT/IG will look specifically at oversight of maintenance practices at one unspecified carrier. Although the DOT/IG memorandum did not identify the carrier, sources say it most probably is American Airlines.

It is not evident when the DOT/IG will complete its audits. However, its look at the FAA’s practices dovetails with the completion on 30 June of the FAA’s own review of airworthiness directive (AD) compliance at the airlines. The FAA was to complete an assessment of air carrier compliance with a sampling of ADs, but as of this writing the results of that review are unknown. What is known is that sampling only 10% of ADs at each carrier may lead to seriously misleading results. For one thing, the FAA certificate managers are free to choose, from among potentially hundreds of ADs, which ones are to be inspected for compliance, allowing for ample leeway to “game” the system to yield “good news” but misleading results. Some ADs require only inspections or placards and do not involve the kind of detailed B737 structural inspections conducted at Southwest Airlines, or the kind of wiring inspections affecting MD-80s at American Airlines.

An FAA official relates that the agency’s review will be completed by mid-July.

The DOT/IG review, begun 2 June, is, according to the memorandum, for three purposes:

“Congressionally requested follow-up review of ATOS [Air Transport Oversight System]: The objective of this review is to determine whether FAA has: (1) completed timely inspections of air carriers’ systems for monitoring critical maintenance programs, (2) tested and validated that these carrier systems are operating effectively, and (3) effectively implemented ATOS for the remaining air carriers regulated under 14 CFR § 121.

“Congressionally requested review of FAA’s special inspection of air carrier AD compliance. The objective of this review is to evaluate FAA’s two-part, special-emphasis inspection of air carriers’ compliance with ADs. Specifically, we will evaluate whether FAA: (1) selected a representative sample of ADs for review, (2) thoroughly review air carriers’ compliance with the selected ADs, and (3) ensure that air carriers implemented corrective actions to improve AD compliance where needed.

“Review of FAA’s oversight of maintenance practices at one carrier: The objective of this review is to determine whether: (1) maintenance-related incidents, such as in-flight emergencies and diversions or other negative maintenance-related trends, have increased at this air carrier and (2) FAA has verified that the air carrier’s maintenance program, including its continuing analysis and surveillance, is operating effectively.”

The “continuing analysis and [safety] surveillance” is a reference to the CASS program, which airlines are supposed to have in place to monitor maintenance trends that affect safety. When the jackscrew failed on the Alaska Airlines MD-80 tail in January 2000, and the FAA threatened to ground the airline after the fatal crash, deficiencies in the airline’s CASS were cited as probably the foremost item to be corrected at Alaska Airlines.

Note that the DOT/IG is going to review the mix of ADs selected at each carrier to assure that they constitute a representative sampling. What the DOT/IG apparently does not plan to do is evaluate whether a sampling of 10 ADs in phase one and 10% of ADs in phase two, ending 30 June, is as valid as, say, 50% or 100% of all ADs. It is not clear how the DOT/IG will measure this effectiveness: possible methods include a maintenance records check for AD compliance, a verification of actual AD work on the airplane, or a combination of the two.

One of the ideas propounded by this publication is for the FAA’s principal maintenance inspector (PMI) to sign off on all ADs for timely compliance (see Aviation Safety & Security Digest, home page, ‘Agency Accuses Airline of Sloppy Maintenance While Sidestepping Its Own Safety Oversight Shortcomings’). Since FAA inspectors apparently missed AD anomalies at American Airlines, this alternative at least answers the larger question: “If pre-eminent operators like American Airlines cannot be trusted to reliably complete ADs, is sign-off by the PMI the only reliable alternative?”

The DOT/IG memorandum does not indicate that a serious examination will be made of the FAA’s service difficulty report (SDR) system. Such an examination would provide a heads-up for malfunctions resulting from AD noncompliance, partial compliance, or even those cases where the AD was fully and properly complied with, yet a malfunction still occurred.

If the DOT/IG were to look at the SDR database, it would find that some airlines rigorously file reports; others submit SDRs only occasionally, while a few carriers do not submit them at all. Because of this scandalously inconsistent reporting discipline, the SDR database cannot be mined for system wide trends.

Indeed, the National Transportation Safety Board (NTSB) has complained four times about deficiencies to the SDR database:

83 June 1993, to the FAA: “Attempts to effectively use the SDR data base in recent Safety Board investigations have revealed that the current program is incomplete and of limited value in identifying accurate service defect histories because many reportable service difficulties are not reported to the FAA.

    “This situation was identified during the Board’s investigation of an accident involving the failure of a Cessna 208 landing gear shimmy damper and cracking of the engine/nose gear mounting structure. The FAA SDR data base revealed only two reports of engine mount cracking and no prior reports of shimmy damper failure. However, data supplied by the airplane manufacture showed 17 reports of engine mount cracking and 250 reports of shimmy damper failure. …

     “The GAO and the IACWG [International Airworthiness Working Group] detailed many shortcomings of the SDR system …

“(1) The SDR system only contains a small percentage of the actual occurrences;

(2) Vagueness in reporting requirements and airlines’ concerns about public access to SDR data contribute to low SDR reporting;

(3) Doubts about the system’s capabilities and effectiveness have discouraged SDR reporters, users, analysis;

(4) The service and safety data maintained by the manufacturers are more useful, comprehensive, and timely than the FAA’s SDR data;

(5) FAA analysis of SDRs occurs rarely or not at all; and

(6) FAA staff limitations and management inattention contribute to SDR program ineffectiveness.”

89 January 1998, to the FAA: “Relate to the operators who submit SDR the need for complete and accurate information [and] remind FAA inspectors … of their need to review the component failure reports for accuracy and completeness.”

811 September 2002, to the FAA: “Exposure data are not adequately detailed to support the analysis of risk factors. …

     “When investigating a 1992 accident in New York involving a Trans World (TWA) L-1011, the Board searched the FAA’s SDR data base for information on stall warning system failures. The database contained only 2 such reports, but 24 additional records were identified by Lockheed and TWA.”

811 October 2002, letter to the Bureau of Transportation Statistics, Dept. of Transportation: “The Safety Board’s ability to study important safety issues is often affected by poor data quality.”

The NTSB’s pleadings have had no effect on improving the usefulness of the SDR database. In June, the FAA took the database offline with the claim that it was “hacked.” That assertion has to be measured in light of the fact that since 4 June the database has been unavailable to the worldwide aviation community.

Finally, a few comments are in order about ATOS. Introduced in 1998 after the ValuJet crash of 1997, the system was publicized by the FAA as replacing the need to inspect airlines and airplanes with the studied approach of searching through records and documents to identify trends that could threaten safety. Through this type of trend analysis – rather than actually inspecting aircraft – the FAA hoped the airlines would be able to assess themselves, and FAA inspectors would only have to watch for indications of breakdowns in airline self-management.

At least two problems pertain to ATOS as conceived. First, ATOS only applies to regularly scheduled airlines. Many cargo planes, passenger charter and business jets are not covered. Thus, the “one level of safety” ethic pushed by the FAA does not apply when it comes to one of its main oversight programs, which has been implemented only at a fraction of operators. Second, the maintenance problems found at Southwest Airlines and American Airlines were discovered by actually looking at airplanes. A records check revealed only part of the full story and only partially uncovered the scope of AD noncompliance.

One should note that with the SDR inoperative, ATOS is about all that’s left to assess the safety of a carrier’s operation. But the Government Accountability Office (GAO) recently raised questions about the effectiveness of the ATOS program. The FAA’s response to this report was telling, as it characterized the GAO’s assessment as “too negative.”

These caveats aside, the FAA is nevertheless facing a wide-ranging DOT/IG investigation that may well uncover new problems that, when added to the findings of recent Congressional hearings, could portend wholesale corrective action.