Research and Analysis Report

Why have we done this report

In the last three decades, Australia has seen a significant growth in the number of amateur-built aeroplanes (aircraft built for personal use from an original design, established plans or kit, which are not entirely built and assembled in a factory).  However, the safety record of amateur-built aircraft in Australia had not been robustly established.

What did this report do

The ATSB investigated the safety history of amateur-built aircraft in Australia through analysis of accident data held in the ATSB’s occurrence database from 1988 to 2010. Comparisons were made between accidents involving amateur-built aircraft and those involving similar factory-built aircraft to help identify whether the rate and types of accidents differed between these two groups of aircraft.

What the ATSB found

Amateur-built aircraft had an accident rate three times higher than comparable factory-built certified aircraft conducting similar flight operations between 1988 and 2010. The fatal and serious injury accident rate was over five times higher in amateur-built aircraft, in particular due to relatively more serious injury accidents.

The pilots of amateur-built aircraft involved in accidents were significantly more experienced overall than factory-built aircraft accident pilots. However, they were significantly less experienced on the aircraft type that they were flying at the time of the accident.

Over half of the accidents were precipitated by mechanical events, which were mainly complete or partial engine failures. Following the amateur-built phase one test period, mechanical failures were still significantly more common when compared with factory-built aircraft. A quarter of accidents were from loss of aircraft control. Structural failures were not common precursors in amateur-built aircraft.

Collision with terrain and forced landing accidents were more frequent in amateur-built aircraft. Collisions with terrain, hard landings, and runway excursions were more likely to result in a serious injury from an amateur-built aircraft accident than for factory-built accidents.

Safety message

Builders of amateur-built aircraft should select, install and maintain aircraft engines carefully as engine issues are the most likely reason why an accident will occur. Careful consideration to occupant protection at the time of building is also encouraged as serious injuries have been disproportionally more common.

Owners of amateur-built aircraft should ensure they have adequate training in the same type of aircraft before operating the aircraft they have built, or purchased second-hand.

Executive summary

The ultimate purpose of a safety investigation is to enhance safety, and it is not the purpose to apportion blame or liability. A safety investigation into an occurrence (accident or incident) can enhance safety by identifying safety issues and communicating these issues to relevant organisations. It can also enhance safety by providing information about the circumstances of the occurrence and the factors involved in the development of the occurrence to the transportation industry. 

The quality of a safety investigation’s analysis activities plays a critical role in determining whether the investigation is successful in enhancing safety. However, safety investigations require analysis of complex sets of data and situations where the available data can be vague, incomplete and misleading. Despite its importance, complexity, and reliance on investigators’ judgements, analysis has been a neglected area in terms of standards, guidance and training of investigators in most organisations that conduct safety investigations. 

To address this situation, the Australian Transport Safety Bureau (ATSB) developed a comprehensive investigation analysis framework. The framework consists of: 

• a defined process or workflow for conducting analysis activities; 
• standardised terminology and definitions; 
• an accident development model (termed the ATSB ‘investigation analysis model’); 
• and policies, guidelines, tools and training for investigators. 

As with all analysis approaches, some concerns have been raised regarding aspects of the ATSB framework, particularly regarding the standard of proof used to determine contribution to the development of an occurrence and the nature of the ATSB investigation analysis model. In terms of standard of proof, the ATSB framework defines a ‘contributing safety factor’ as a safety factor that, if it had not occurred or existed at the relevant time, then either the occurrence would probably not have occurred, adverse consequences associated with the occurrence would probably not have occurred or have been as serious, or another contributing safety factor would probably not have occurred or existed. The term ‘probably’ was defined as being equivalent to ‘likely’ and meaning more than 66 per cent likelihood (a definition used by the Intergovernmental Panel on Climate Change). 

Because of its focus on future safety, the ATSB definition adopts a ‘link-by-link’ approach, where the judgement about whether a safety factor contributed to the development of an occurrence is made in terms of its relationship to another contributing safety factor. In contrast, other types of investigations (particularly those whose purpose is to determine responsibility) generally use a ‘relative-to occurrence’ approach. With the relative-to-occurrence approach, judgements of contribution are made in terms of the safety factor’s relationship to the occurrence itself. The ATSB analysis framework will involve a higher standard of proof than in Australian coronial inquests or civil legal proceedings for factors relatively close in proximity to the occurrence (that is, more than 66 per cent versus more than 50 per cent). But as an ATSB safety investigation proceeds to identify contributing safety factors more remote from the occurrence, the degree of relationship of the factors to the occurrence itself will generally decrease using the ATSB framework. 

Associated with the concept of standard of proof is the concept of standard of evidence, or the quantity or quality of evidence required before a decision maker can be satisfied that the relevant standard of proof has been met. In the Australian legal system, the ‘Briginshaw scale’ is used when making judgements about the standard of evidence. The scale involves considering the seriousness of a finding, the inherent unlikelihood of a finding, and the gravity of consequences that flow from a finding for the party or parties involved. The Briginshaw scale is not routinely incorporated into safety investigation methods. The scale is used to some extent in ATSB safety investigations, but there are several reasons to consider that it is not required or beneficial for the ATSB to apply the scale more broadly. 

The differences between the ATSB approach to determining contribution and other approaches may be a matter of nuance in many situations, and similar findings may result regardless of the approach being used. Nevertheless, there is also the potential for different sets of findings to be produced. More specifically, the ATSB’s link-by link approach together with a ‘probable’ standard of proof has the following advantages over many other investigation analysis approaches: 

• It better enables the search for potential safety issues, particularly those more remote from an occurrence. The enhanced searching will result in more safety issues being identified and communicated to relevant organisations to enhance safety. 
• It has greater potential for providing a richer or more detailed description of the factors involved in the development of an occurrence, which provides better learning opportunities for the transport industry.  
• It is more distinct from the approach used in legal proceedings for determining blame or liability. Therefore, there is less potential for the existence of barriers to learning or safety action due to an investigation’s findings being associated with such legal proceedings, or interpreted with such proceedings in mind. 

In terms of the ATSB investigation analysis model, it is based on the widely used Reason model of organisational accidents and consists of five levels of safety factors (occurrence events, individual actions, local conditions, risk controls and organisational influences). Concern has been raised that the model is biased towards finding problems at the higher levels of the model and that the pendulum has swung too far towards searching for organisational factors. The ATSB model does encourage investigators to look for problems with risk controls and organisational influences as, if there are problems in these areas, this is where significant safety enhancements can be made. However, the model is only used as one means to help identify potential safety factors. Before any findings are made about whether these potential factors contributed to the development of the occurrence, or were otherwise important, they need to be tested or verified. In the ATSB analysis framework, this involves using a structured process to examine the available evidence and conducting tests for existence, influence and importance. 

The ATSB believes that its investigation analysis framework is well suited to its role as an independent, no-blame safety investigation body. It is hoped and expected that ongoing development and provision of information about the framework can help the safety investigation field as a whole consider some important issues and help develop the best means of conducting safety investigations to enhance safety. Accordingly, any feedback or comment that any individual or organisation has regarding the ATSB analysis framework, ways to enhance the framework, ways for the ATSB to better communicate its findings, or any other matters discussed in this report would be gratefully received.

The purpose of this report was to examine the relationship between ageing aircraft and flight safety, to determine the chronological age of the Australian aircraft fleet, and to review current and future directions for the management of ageing aircraft.

Age can be managed by retiring the aircraft and purchasing a newer aircraft or through adequately maintaining ageing aircraft through additional and specific maintenance. This requires cooperation between regulators, manufactures, maintainers, operators, and owners. Continuing airworthiness programmes and Supplementary Inspection Programmes are methods of ensuring adequate maintenance. Ageing of an aircraft can be a safety issue, but with adequate maintenance, the consequences of ageing can be mitigated. Current and future maintenance programmes will act as a preventative measure to reduce the safety risk associated with ageing aircraft, but only if the operators adhere to the programmes.

In Australia, the average age of fleet of turbofan aircraft is low, and has been decreasing. Multi-engine turbofan aircraft with a maximum take-off weight between 50,001 and 100,000 kg had the lowest average age in 2005 at just 6 years. This was the only aircraft category whose average age decreased over the period 1995 to 2005. The turbofan aircraft with a maximum take-off weight of more than 100,000 kg had an average age of 11 years in 2005. The high-capacity turbofan aircraft receive extensive continuing airworthiness support from the manufacturers. The low age and extensive continuing airworthiness support provide a double defence to ensure the safety of the Australian multi-engine turbofan aircraft fleet.

The piston engine fixed-wing aircraft fleet, by contrast, had the highest average age at 30 years. These aircraft often do not receive the same level of continuing airworthiness support from the manufacturer as the turbofan aircraft. In Australia, multi-engine piston aircraft are often used in regular public transport and charter operations, and therefore the high average age needs to be considered in relation to their safe operation in passenger services.

Alcohol is a widely used drug, and its abuse is a serious public health problem. Alcohol has many widespread effects on the body, and impairs almost all forms of cognitive function, such as information processing, decision-making, attention and reasoning. Visual and vestibular functions are also adversely affected. The performance of any demanding task, such as flying an aircraft, is thus impaired by the effects of alcohol. Many studies have shown a significant proportion of aircraft accidents associated with alcohol use. Alcohol increases the risk of spatial disorientation, hypoxia and poor +Gz tolerance. Many studies have consistently shown significant detrimental effects of alcohol on pilot performance, both in the acute stages and in the post-alcohol period for up to 48 hours. Even low doses of alcohol can lead to reduced performance. While a pilot may legally fly 8 hours after drinking, the residual effects of alcohol may seriously impair their performance, especially in high workload and demanding situations.

Fuel exhaustion and fuel starvation accidents continue to be a problem in the Australian aviation industry, accounting for over 6 per cent of all accidents between 1991 and 2000. 

Within Australia, fuel exhaustion refers to those occurrences where the aircraft has become completely devoid of useable fuel. Fuel starvation refers to those occurrences where the fuel supply to the engine(s) is interrupted, although there is adequate fuel on board the aircraft. 

The current study investigates the overall rates of factors contributing to and significance of fuel-related accidents between 1991 and 2000. While fuel starvation accident rates have remained relatively stable over the past 20 years, fuel exhaustion accident rates have shown a significant decrease of 29.6 per cent. 

Between 1991 and 2000, there were a total of 139 fuel-related accidents reported to the Australian Transport Safety Bureau (ATSB). 

As a result, 49 lives were lost, with an estimated cost to the Australian community of between $63 million and $127 million (in 1996 Australian dollars).

Australia has an excellent air transport safety record. Major Australian airlines have long been regarded as being among the world's safest, and there have been no fatalities involving an Australian high-capacity jet aircraft. This enviable record is due, in part, to an aviation safety culture that recognises the need for constant safety awareness.

Given the commercial pressures facing international aviation, the events described in this report should be seen as a learning experience for the aviation industry, regulatory bodies, and all organisations concerned with continuing airworthiness assurance.

In December 2000 and in April 2001, a number of Ansett Australia (Ansett) Boeing 767 (B767) aircraft were withdrawn from service because certain required fatigue damage inspections of the aircraft structure had been missed. As a result there was uncertainty as to the continuing airworthiness status of the aircraft. In December 2000 the concerns related to possible fatigue cracking in the rear fuselage of the aircraft, and in April 2001 the concerns related to possible fatigue cracking of the engine strut fitting on the wing front spar.

On 11 January 2001, the Australian Transport Safety Bureau (ATSB) commenced an investigation into the circumstances surrounding the withdrawal from service of the Ansett B767 aircraft as the situation was regarded as indicative of a potential safety deficiencyⁱ. On 10 April 2001 the ATSB investigation was extended to include an examination of the continuing airworthiness system for Australian Class Aⁱⁱ aircraft such as the B767.

Action by Ansett and the Civil Aviation Safety Authority (CASA) addressed the potential risks to fare-paying passengers. Although Ansett was subsequently placed into voluntary administration in September 2001, the ATSB continued a detailed systemic investigation because of the importance of the issues involved, both in Australia and internationally.

The international continuing airworthiness system, like all complex and safety-critical activities, is dependent on robustⁱⁱⁱ systems to maintain high reliability. The circumstances surrounding the withdrawal from service of the Ansett B767 aircraft revealed, among other things, that the reliability of the continuing airworthiness system was threatened by a number of weak defences.

The B767 aircraft type was among the first in the world to be designed and certified under damage tolerance principles. Damage tolerance certification relies heavily on scheduled inspections to ensure continuing airworthiness. The aircraft structure is designed to maintain integrity until any fatigue or corrosion damage can be detected at a scheduled inspection, and appropriate action taken. Therefore, in itself, the presence of fatigue cracks in the Ansett B767 aircraft was not necessarily a cause for undue concern. However, it was critical that there were robust systems to ensure that the required structural inspections were carried out to detect the cracks before they exceeded acceptable limits.

Withdrawal from service of Ansett B767 aircraft in December 2000

Ansett was the sixth airline worldwide, and the first airline outside North America, to operate the B767. Of the nine Ansett B767-200 aircraft, five were first flown in 1983 and two in 1984. The aircraft accumulated a high number of flight cycles^iv because they were mostly flown on comparatively short domestic sectors. Ansett had been working with Boeing on fatigue cracking in the area of the Body Station^v 1809.5 bulkhead outer chord since 1996.

In June 1997, Boeing introduced the Airworthiness Limitations Structural Inspection program^vi for the B767. The program was an essential part of the damage tolerance requirements and was designed to detect fatigue cracking in susceptible areas that had been identified through testing and in-service experience. Ansett staff did not initially recognise that some Airworthiness Limitations Structural Inspections were required by 25,000 cycles and a period of almost two and a half years elapsed before that error was identified. At the time that the inspection program was introduced, some Ansett B767 aircraft had already flown more than 25,000 cycles. In June 2000, further 25,000 cycle inspections were introduced, including in the area of the Body Station 1809.5 bulkhead outer chord. Ansett did not initially act on this.

In December 2000, Ansett senior management became aware of the missed inspections and the aircraft were withdrawn from service on 23 December 2000, despite the high commercial cost to the company. At that time, both Ansett and CASA were of the belief that compliance with the missed inspections was mandatory. Subsequent legal advice indicated that the regulatory basis for mandating compliance with the Airworthiness Limitations Structural Inspections for Australian operators was unclear. On 29 December 2000, CASA issued a direction to Ansett specifically mandating the inspections for the Ansett B767 aircraft.

The ATSB investigation found that the Ansett system for the introduction and scheduling of the B767 Airworthiness Limitations Structural Inspections was deficient and vulnerable to human error. A mistake or omission by one or two people could potentially result in continuing airworthiness assurance being compromised. In addition, deficiencies existed in resource allocation and in the supporting information management systems.

From October 1998, Boeing also issued a series of service bulletins in relation to fatigue cracks in the area of the B767 Body Station 1809.5 bulkhead outer chord. Service bulletins are issued by aircraft, component, or engine manufacturers to provide operators with relevant service information. Not all service bulletins are safety-related, and compliance with a particular service bulletin can only be mandated by the State of Registry of an aircraft.

Boeing initially notified operators that the service bulletin requirements were primarily commercial in nature. It was not until November 2001 that Boeing indicated that the service bulletin dealt with a potentially major safety issue. The FAA had mandated action by US operators in relation to the service bulletin in April 2001.

Any action to be taken by Ansett in relation to the Body Station 1809.5 service bulletins issued by Boeing was complementary to requirements under the B767 Airworthiness Limitations Structural Inspection program. It was the failure by Ansett to appropriately incorporate the required Airworthiness Limitations Structural Inspections, issued in June 1997 and updated in June 2000, into the B767 system of maintenance that led to the withdrawal from service of six Ansett B767 aircraft in December 2000.

Withdrawal from service of Ansett B767 aircraft in April 2001

In March 2000, Boeing issued an Alert^vii service bulletin to detect and repair fatigue cracks in the wing front spar outboard pitch load fitting of the B767 engine mounting strut. Boeing recommended that the work be carried out within 180 calendar days. A revision to the service bulletin was issued in November 2000. In March 2001, Ansett became aware that they had not acted on either the original or the revised service bulletins.

During the period from 7-9 April 2001, inspections revealed cracks in the pitch load fittings of three of the Ansett B767 aircraft and they were withdrawn from service. On 9 April 2001 CASA required that a further four Ansett B767 aircraft be withdrawn from service, pending inspection. Those inspections were subsequently carried out, and the aircraft were cleared to fly.

Deficiencies in the Ansett engineering and maintenance organisation

The ATSB investigation found that similar deficiencies within the Ansett engineering and maintenance organisation led to the withdrawal from service of the B767 aircraft in December 2000 and April 2001. Those deficiencies were related to:

• organisational structure and change management
• systems for managing work processes and tasks
• resource allocation and workload.

However, the investigation found no evidence to suggest that Ansett had deliberately breached airworthiness regulations.

Ansett had undergone considerable change over a number of years. Many of the Ansett systems had developed at a time when the company faced a very different aviation environment. Over time, efficiency measures were introduced to improve productivity but the introduction of modern robust systems did not keep pace with the relative reduction in human resources and loss of corporate knowledge.

Risk management and implementation of change within the Ansett engineering and maintenance organisation were flawed. Inadequate allowance was made for the extra demand on resources in some key areas during the change period.

The Ansett fleet was diverse and the point had been reached where some essential aircraft support programs were largely dependent on one or two people. Hence it was possible for an error or omission by a particular specialist to go undetected for a number of years.

Resource allocation and workload issues had been evident within some areas of the Ansett engineering and maintenance organisation for a considerable period of time. The investigation found that measures aimed at achieving greater productivity had been introduced throughout the organisation without sufficient regard to the different circumstances and criticality of the different work areas. Insufficient consideration had been given to the possible consequences of resource constraints on the core activities of some safety-critical areas of the organisation.

People and robust systems are two of the prime defences against error. Therefore, a combination of poor systems and inadequate resources has the potential to compromise safety. If a failure by one or two individuals can result in a failure of the system as a whole, then the underlying problem is a deficient system, not simply human fallibility.

The Australian continuing airworthiness system

The ATSB investigation found that based on the Ansett B767 experience, the Australian system for continuing airworthiness of Class A aircraft was not as robust as it could have been, as evidenced by:

• uncertainty about continuing airworthiness regulatory requirements
• inadequate regulatory oversight of a major operators continuing airworthiness activities
• Australian major defect report information not being used to best effect.

The investigation identified a need for the regulatory basis for continuing airworthiness requirements of Class A aircraft to be better defined and disseminated to operators.

No evidence was found to indicate that CASA had given formal consideration to monitoring the introduction of the B767 Airworthiness Limitations Structural Inspection program by Ansett from 1997 onwards.

Prior to December 2000, there was apparently little or no awareness among Ansett senior management or within CASA of the underlying systemic problems that had developed within the Ansett engineering and maintenance organisation. The presence of organisational deficiencies remained undetected. In addition, there were delays in adapting regulatory oversight of Ansett in response to indications that Ansett was an organisation facing increasing risk.

The decision by the then Civil Aviation Authority in the early 1990s to reduce its previous level of involvement in a number of safety-related areas did not adequately allow for possible longer-term adverse effects. This included reducing the work done by Authority specialist staff in reviewing manufacturer's service bulletins relevant to Australian Class A aircraft, and relying on operators' systems and on action by overseas regulators in some airworthiness matters.

CASA's central database for major defect reports was incomplete, partly due to deficiencies in reporting, and the information received was not always fully analysed. In addition, feedback to the initiators of major defect reports, and to other operators, was limited. As a result, the potential safety benefit of the major defect reporting system was not fully achieved.

The FAA and ICAO

Delays by the US Federal Aviation Administration (FAA) contributed to a lack of awareness by Ansett and CASA of required B767 Airworthiness Limitations Structural Inspections. This breakdown in FAA process was acknowledged by the US Secretary of Transportation in August 2001. The FAA did not issue airworthiness directives in relation to the June 1997 Airworthiness Limitations Structural Inspection program, or the service bulletins for the Body Station 1809.5 bulkhead outer chord and the wing front spar outboard pitch load fitting, until after the second Ansett groundings in April 2001.

Different views within the FAA as to the importance of airworthiness directives to mandate continuing airworthiness requirements for damage tolerance aircraft types contributed to a lack of timely action by the FAA. The ATSB report includes recommendations that the FAA ensure that such airworthiness directives are processed and released without undue delay, and that affected parties should be informed when delays do occur. The report also recommends that the FAA ensure that the process for determining grace periods for aircraft to comply with airworthiness directives is both systematic and transparent.

The ATSB report outlines where the existing international continuing airworthiness system, as defined by International Civil Aviation Organization (ICAO) standards and recommended practices, could be enhanced by the application of quality assurance mechanisms to the processing and distribution of safety-related information.

The events outlined in this report indicate that there was a breakdown in the continuing airworthiness system within Ansett, the FAA, and CASA. In addition, the possible safety significance of cracks in the area of the B767 Body Station 1809.5 bulkhead outer chord was not initially highlighted by Boeing.

Safety action

On 12 April 2001, the ATSB released two safety recommendations to CASA. The intent of these recommendations was to enhance the robustness of the systems used to manage the continuing airworthiness of Australian registered aircraft such as the B767 by ensuring that:

• action, or lack of action, by another State did not adversely affect the safety of Australian Class A aircraft
• all service bulletins relevant to Australian Class A aircraft were received, assessed and implemented or mandated as appropriate.

CASA subsequently initiated a comprehensive review of its systems to monitor, assess, and act on service bulletins, to ensure that those critical to safety could be readily identified and acted upon appropriately. Recommendations from that review were addressed in an associated implementation plan that detailed the nature and timing of the actions that CASA would take in response to the recommendations. The ATSB is monitoring the implementation of this important safety action.

In response to the circumstances of the events of December 2000 and April 2001, the FAA has included further checks and balances designed to ensure that all service bulletins issued by US manufacturers are properly reviewed and addressed. In addition, the FAA has established an 'early warning system' to provide non-US airworthiness authorities with information on pending occurrence investigations that may result in mandatory action by the FAA.

The manner in which events developed highlights the need for organisations to be continually mindful of potential threats to safe operations. Periodic review is needed to ensure that existing systems for maintaining air safety keep pace with the changing environment.

Implementation by the relevant organisations of the recommendations made by the ATSB as a result of this investigation should help to ensure that aviation systems, both within Australia and internationally, are strengthened and that air safety for Class A aircraft is enhanced.

i Section 19AD of the Air Navigation Act 1920 defines a safety deficiency as any situation related to aviation that can reasonably be regarded as having the potential to affect adversely the safety of aviation.

ii Class A refers to an aircraft with a Certificate of Airworthiness issued in the transport category, or one that is used for regular public transport operations.

iii In the context of this report, a system is robust if, when someone makes an error or a problem occurs for some other reason, the system can detect the deviation and recover without any significant negative effect.

iv A flight cycle is one completed take-off and landing.

v Body Station number is the distance in inches from a datum in front of the nose of the aircraft to a particular point of the aircraft structure.

vi The Airworthiness Limitations Structural Inspections were in addition to zonal inspections and scheduled structural inspections that had formed part of the B767 maintenance program from the time the aircraft entered service.

vii Boeing service bulletins are classified into three categories in order of urgency: Alert, Unusually Significant, and Standard. Alert service bulletins are issued for safety-related issues that require the immediate attention of the operator.

On the 20th May 1988 at approximately 1609 hours, a Cessna 172 collided with a Piper Tomahawk in the circuit area at Coolangatta, Queensland. The accident, in which four people died, occurred in conditions of good visibility.

This collision and others which occurred in the late 1980s drew attention to the deficiencies of the see-and-avoid concept.

The Coolangatta accident report stated that: 'As a result of this accident, the Bureau of Air Safety Investigation has undertaken to conduct an evaluation and prepare a report on the practicability of the see and be seen (see-and-avoid) principle in controlled and non-controlled airspace.' (BASI report 881/1042).

This report, prepared in response to that undertaking, summarises the research relevant to unalerted see-and-avoid and is intended as a reference document for Civil Aviation Authority (CAA), Industry, and BASI personnel as well as a source of recommendations. The report does not analyse the Australian accident experience.