At 1353 Eastern Standard Time, a Saab 340 departed Sydney for Tamworth climbing to flight level (FL) 140. The flight was conducted within controlled airspace and crossed the boundary between the Sydney and Brisbane Flight Information Regions (FIR), 45 nm north of Sydney. The airline had chosen to use the aircraft registration instead of the flight number as the callsign. This flight stage was one of six separate flight plans held within The Australian Advanced Air Traffic System (TAAATS) for the aircraft. The flight plans had varying departure times, two of which were for Sydney-Tamworth flights. Companies that chose to use flight numbers as the aircraft callsign would have had one unique flight data record for each flight stage held within TAAATS.

TAAATS used a messaging system for flights that cross the FIR boundary, this system used messages called ASIA/PACIFIC ATS Inter-Facility Data Communications (AIDC). The normal AIDC system messaging occurred between the Melbourne and Brisbane Flight Data Processors (FDP). However, as there were several plans in the system for the Saab 340, the Brisbane FDP did not send an accept message (ACP) to the Melbourne FDP. In the absence of an ACP from Brisbane, the Melbourne FDP created an unsuccessful coordination "U" warning in the Sydney controller's label and an aural alarm was generated, indicating that coordination message exchange between Brisbane and Melbourne FDP's was unsuccessful.

The unsuccessful (undeliverable) AIDC coordination messages between the two FDP's were automatically sent to the BN Flight Data Coordinator (TFDC). The Manual of Air Traffic Services (MATS) required that the TFDC print these messages and give them to the controller concerned, however on this occasion the TFDC did not pass on these messages.

The Sydney controller received and acknowledged the "U" warning and alarm. Once the alarm had been acknowledged, the system removed the "U" warning from the screen and the controller then had no visual indication that the aircraft had been the subject of an unsuccessful coordination alarm.

MATS required that aircraft that are subject to a "U" alarm are to be verbally coordinated to the sector in the other FIR. The voice coordination for the Saab 340 did not occur. The alarm was also not mentioned by the Sydney controller during a hand-over to another Sydney controller.

The second Sydney controller initiated the electronic radar hand-off on the Saab 340 to the Brisbane controller. Without waiting for formal acceptance from the Brisbane controller, the Sydney controller instructed the crew to contact Brisbane Air Traffic Control. This was not in accordance with procedures.

The crew of the Saab 340 attempted to contact the Brisbane Maitland (MND) sector controller three times. During the first transmission, the controller was conducting coordination with another unit, and did not hear the aircraft. The second time the crew attempted to contact MND, another pilot over transmitted and the controller did not hear the transmission. The third time, the crew only used the callsign of the aircraft and did not advise their flight level. The controller instructed the crew to "standby".

The MND controller was under training, and had only just returned to controlling duties after a long absence. The MND controller did not call the crew of the Saab 340 back.

As the Brisbane FDP had not processed the flight data record (FDR) for the Saab 340, the aircraft appeared on the MND air situation display (ASD) as a black "unconcerned" track. This track would only have displayed a secondary surveillance radar code and level in the label. It did not display the aircraft's call sign.

When the Saab 340 entered Brisbane's airspace 45 nm north of Sydney, there were numerous other black tracks outside controlled airspace (OCTA) on the MND sector ASD.

At the time the crew of the Saab 340 were attempting to contact the MND controller, the aircraft's label was superimposed on another black track directly underneath it. It is a common occurrence to have aircraft labels overlying each other. TAAATS provides functionality to enable the controllers to move the aircraft labels to reduce label clutter.

The aircraft then passed through Nambucca (NAA) and Armidale (ARM) Sectors, as a black track in controlled airspace, without being detected by the controllers.

Recorded radar data indicated that the Saab 340 did not pass adjacent to any other aircraft and consequently there was no infringement of separation standards. The first time that the Armidale (ARL) sector controller became aware of the aircraft was when the crew of the Saab 340 eventually contacted the MND controller for descent into Tamworth. The aircraft was subsequently radar identified and processed normally into Tamworth.

The pilot of the Bell Jet Ranger reported that while cruising at about 2,500 ft AGL, the rotor RPM indication decreased to zero. He initiated auto-rotation immediately by lowering the collective pitch. The controls were very stiff, however all engine parameters were normal. The helicopter descended at higher than normal speed and landed heavily. The right skid crosstube support failed and the aircraft rolled onto its side. Both occupants vacated the wreckage with minor injuries.

The pilot reported that he did not hear the low rotor alarm and did not notice the low rotor RPM caution light illuminate.

Investigation determined that the drive spline and coupling to the hydraulic pump and rotor tachometer generator were devoid of lubrication and worn to the degree that drive to both units had failed. This resulted in the loss of hydraulic power to the controls and the loss of the main rotor speed indication.

The low rotor RPM sensor which provides a signal to activate the low rotor alarm and the low rotor RPM caution light was tested and functioned normally. However, it is likely that the pilot did not notice the low rotor RPM warnings, due to his immediate initiation of an auto-rotation, and the ambient light conditions. The warning horn is cancelled by a cut-out switch on the collective pitch handle when the handle is lowered as in the case of an auto-rotation. As the helicopter was tracking south at the time, the bright sun from the northeast would have made illumination of the low rotor RPM caution light difficult to distinguish.

Investigation of the maintenance requirements for the helicopter found that, prior to January 1998, the hydraulic pump drive splines were required to be lubricated by hand every 1200 hours or twelve months, whichever came first, or under extreme operating conditions, every 300 hours. In January 1998 the manufacturer issued a revised manual with a changed requirement for the Bell 206B III model to lubricate the hydraulic pump drive splines every 300 hours with no calendar requirement.

The accident helicopter had only accumulated about 90 hours time in service since January 1998 and 217 hours since the initial issue of its Australian certificate of airworthiness in 1990.

Sequence of events

The Sydney Terminal Control Unit (TCU) provided air traffic services within 45 NM of Sydney Airport, to a height of 28,000 ft. The normal power supply for the TCU consisted of mains or generator supply feeding into two separate, independent Uninterruptible Power Supply (UPS) units consisting of "A" and "B" systems that shared the electrical load.

A routine 6 monthly performance inspection of The Australian Advanced Air Traffic System (TAAATS) Sydney TCU UPS was scheduled from 1500 Eastern Standard Time on 6 July until 1800 the next day, under an approved works plan. This performance inspection was conducted ahead of schedule to avoid the Olympics period.

The electrical technical officers scheduled to conduct the inspection were delayed because of higher priority tasking associated with the Sydney control tower. Subsequent approval to start the inspection at 1800 was gained under normal procedure through the Melbourne Technical Customer Interface and the Sydney TCU Traffic Manager. The approval task of the TCU traffic manager in allowing works plans to go ahead was not based on any structured risk assessment process. The approval was based on the experience of the traffic manager in assessing and forecasting aircraft movements, staff availability and knowledge of the likelihood and outcome of failure of the power supply for the TCU. Information contained in the works plan that there would be no interruption to service, combined with the stability of the power supply during past similar works, was used to support the traffic manager's decision to approve the works.

Work started on the performance inspection at 1800:36. At 1822:24 the Sydney TCU sustained a total loss of electrical power. Power was restored at 1822:38, 14 seconds later.

This loss of electrical power caused TCU Air Traffic Control (ATC) workstations, software switching of voice communications channels, satellite communications, provision of the Sydney Terminal Approach Radar to Melbourne and Brisbane and operational room lighting, to fail.

The ATC workstations automatically began rebooting after the initial 14 second power outage. The workstations were not available for about 7 to 10 minutes longer while they rebooted.

The air traffic controllers in the TCU were unable to determine the relative positions of aircraft under their jurisdiction for about 7 to 10 minutes. By using the emergency bypass air/ground radio, controllers were able to direct flight crews to keep a visual lookout for aircraft and to turn on their Traffic Alert and Collision Avoidance System (TCAS).

Power was not lost in the Sydney Control Tower and all systems continued to operate normally except the controller workstations that went into a "degraded mode" following the initial power outage, however the Sydney Tower controllers were able to provide normal ATC services to aircraft under their jurisdiction.

Full radar display from the Terminal Area Radar was available on the Tower Data Processing and Display System.

The Brisbane and Melbourne TAAATS Centres provided radar services and limited support during the period that the Sydney TCU ATC workstations were not available.

A review of the air traffic control data recorded at the TAAATS centre in Melbourne, showed that there was no infringement of separation standards.

Works plan

The works plan mentioned that an inspection would be carried out between 1500 on 6 June and 1800 on 7 June on the Sydney TCU power supply. The works plan wrongly referred to the month as June instead of July. The works plan stated that during the work there would be one UPS and generator set available and there would be no interruption to the power supply.

UPS

The two UPS units at the Sydney TCU had been upgraded from a 6 pulse system to a 12 pulse system in February 1999 and had been reported as being stable and having not caused any interruption to service since commissioning.

The two separate, independent Uninterruptible Power Supply (UPS) units consisted of "A" and "B" systems that normally shared the electrical load for the TCU. The performance inspection needed the full electrical load of the TCU to be placed on the "B" system while the "A" system was being tested off-line. The electrical technical officers reported the "B" system was switched from mains bypass to take the full load of the TCU at which point power to the TCU was lost. They also reported that all indications on the "B" system were normal, except that the output currents indicated zero.

Subsequent independent testing of the UPS equipment by Airservices and the UPS manufacturer found the system performed satisfactorily. Exhaustive testing of a similar UPS at the manufacturer's testing facility could not reproduce a power loss similar to that experienced by the Sydney TCU.

During testing by Airservices there was an inconsistency identified between the Rectifier Current Limit settings of the "A" and "B" systems. The "B" system rectifier current limit setting was set at 0.6V while the "A" system had a higher setting of 1.945V. The lower rectifier current limit setting of the "B" system may have occurred during training on the "B" system in May 2000. Airservices initial investigation found this setting may have caused the power outage. However, further investigation determined the lower rectifier current limit setting of the "B" system, by itself, would not have caused the power outage to the TCU.

The state of various conditions within each UPS unit is monitored and recorded independently by the National Technical Monitoring System. There are inconsistencies between these records and the recollection of site staff for the actions immediately prior to the power loss. The recorded data is not sufficiently comprehensive to allow the conclusive determination of the cause of the power loss.

Radar

Brisbane and Melbourne TAAATS centres operated normally and kept radar services during the power outage but lost the supply of Sydney Terminal Area and the Mount Boyce (Blue Mountains west of Sydney) radar data.

The Sydney TCU "fallback" radar system that was designed to assist controllers in maintaining situational awareness in the event of a complete TAAATS failure also failed because it was powered by the same power supply that was subject to the power outage.

Display of the Sydney Terminal Area Radar was available in Sydney Tower on the Tower Data Processing and Display System.

Communications

Recordings of air/ground and ground/ground communications for Sydney Tower and TCU were not available for 73 seconds from 1821:37 until 1822:51.

The Voice Switch panels at each console failed causing the loss of normal air/ground and ground/ground communications. Bypass air/ground communication remained available at those consoles which use remote Very High Frequency (VHF) outlets (50 volt battery powered communications equipment). Bypass air/ground communication was available within a few seconds after the 14-second power outage. Some controllers tried to use the air/ground bypass equipment during the 14-second outage and found that it was not available. The air/ground bypass equipment was used inconsistently depending on which controllers had tried to use the system within the 14 seconds and which controllers tried to use it later.

Following the power outage TCU controllers broadcast advice of the failure to air crew and advised them to keep a visual lookout and to ensure that their Traffic Alert and Collision Avoidance System (TCAS) was switched on. Some inbound aircraft were transferred to Sydney Tower controllers and outbound aircraft were transferred to Melbourne and Brisbane TAAATS centres.

There was no air/ground bypass equipment fitted to the Sydney TCU Directed Traffic Information position.

The individual voice switch restored to a ground/ground bypass mode that allowed controller's access to the PABX. However, that system was not used because the secondary display windows that stored the telephone numbers were not accessible.

The Voice Switch Management Station that controlled restarting of the Voice Switch had an electronic latching switch that activated during the initial power outage. This needed a radio technician to manually reset a switch that allowed the Voice Switch to automatically reboot. The voice switch was not available for TCU controller use until 1832.

Sydney Tower air/ground communications were unaffected. However, ground/ground communications from Sydney Tower controllers to the TCU controllers were unavailable until 1832.

The Airport Rescue and Fire Fighting fire alarm system continued to work without any equipment error recorded.

Sydney TCU supervisor's PABX telephones were available during the power outage using separate handsets and dial pads.

The Sydney TCU Team Leader advised the Sydney Tower Traffic Management Coordinator by telephone that "we've lost everything down here". The Tower Traffic Management Coordinator replied "so have we". The Tower Traffic Management Coordinator did not understand the extent of the failure of the TCU. Consequently, the TCU Team Leader did not request radar support from the Tower Traffic Management Coordinator.

Brisbane and Melbourne TAAATS centres were advised of the power outage by telephone from the Sydney TCU Supervisor's position and by relay of messages from aircraft near Sydney by using the air/ground bypass system.

There was no message on the Computerised Automatic Terminal Information System advising flight crews of the reduced services being provided by the Sydney TCU.

Air traffic control equipment

The 14-second power interruption caused the Sydney TCU TAAATS workstations to automatically reboot. The estimated time for the computers to reboot was between 7 and 10 minutes. This left the controllers without any air situation display for between 7 and 10 minutes.

During the outage, tools were not available for the controllers to maintain situational awareness.

The Sydney Tower air situation display lost new code/callsign correlation but kept the existing code/callsign correlations and automatically went into bypass (a degraded mode) because of the loss of the TCU radar data processor. This limited radar display and the Tower Data Processing and Display System was available for reference by the TCU controllers but was not used because of the misunderstanding of the extent of the equipment outage.

Airspace design

The segregated airspace design used in the Sydney TCU provided adequate short-term protection for aircraft to remain separated during the power outage without any ATC intervention.

The investigation found that for operations at Sydney Airport within the curfew period between 2300 to 0600, there was no airspace segregation between the arrival and departure stages of flight.

The Standard Terminal Arrival Route (STAR) communication failure procedures advise flight crew to track to the Sydney VOR and then fly the most suitable instrument approach to the nominated runway in accordance with the Enroute Supplement Australia (ERSA) emergency section. The investigation found the design of some of the instrument approaches into Sydney precluded flight crew from making an instrument approach from overhead the Sydney VOR during communication failure. As an example, flight crews that were advised to expect runway 34 Right for arrival before the power failure, would track to the VOR and would not be able to start the runway 34 Right ILS approach because this approach requires radar vectors to intercept final approach.

Maintenance documentation

The UPS switchboard is physically set up in a manner that when facing the switchboard, the "A" system, is on the left side and the "B" system is on the right side. This is the exact opposite to the schematic diagram for this UPS system, where the "A" system is on the right side of the diagram and the "B" system is on the left side.

There were two different UPS handbooks available to maintenance services staff. One was the Airservices controlled document which was not current for the equipment fitted at Sydney; the other was an updated handbook issued by the manufacturer during staff training in May 2000. The uncontrolled handbook contained the incorrect rectifier current limit setting. The switching procedure for removing and returning a UPS to service had been revised between the two handbooks.

Airways engineering instructions

The electrical performance inspection was carried out under Airway Engineering Instruction (AEI) AEI-3.4053 issue number 4. The purpose of this AEI is "to define the electrical tasks to be carried out during the performance inspection of the static UPS equipment... installed at Airservices Australia facilities". This AEI is non-prescriptive and is a generic instruction to cover all UPS equipment installed at Airservices facilities.

The document does not clearly define the tasks to be conducted, nor does it refer to the manufacturer's instructions about how to correctly carry out those tasks.

The AEI required the full load of the Sydney TCU to be placed on one UPS system. This removes the redundancy of the normal two independent UPS unit configuration and creates a single point of failure.

ATC procedures

The separation assurance techniques used by the Sydney TCU controllers before the power outage aided in maintaining separation standards during the power outage.

Degraded modes procedures were available in check list form for the TCU controllers. The degraded modes check lists were "designed to enable TAAATS controller's to be able to quickly identify degraded or abnormal system operation and to list the procedures:

• to enable an immediate response to maintain safety
• to be adopted for continued safe operations in the degraded mode
• to be adopted when upgrade to normal operations is available"

The check lists were not designed for nor did they address the multiple failures associated with the power outage. It was expected that the reliability and redundancies incorporated in the system design would preclude a total electrical failure.

The degraded modes check list recommended that "Operators should maintain familiarity with operating in these degraded modes and the need to prioritise actions in accordance with the following:

SEPARATE

• maintain separation by use of alternative standards if necessary
• issue traffic information as required

COORDINATE

• advise others of your degraded status
• adopt full verbal coordination and handoff procedures

UPDATE

• maintain and modify the flight data record (FDR)"

Sydney Tower controllers stopped aircraft departures from Sydney in accordance with degraded mode procedures.

ATC training

Most of the controllers rostered on duty during the power outage had completed their TAAATS conversion training in 1999; some had been more recently trained. The ATC TAAATS training did not cover or simulate the conditions experienced during the power outage.

Refresher training for elements of degraded operation, such as the use of the air/ground bypass system had not been conducted since the early TAAATS conversion training in 1999.

Flight crew procedures

The flight crews involved during the incident followed communication failure procedures as published in the Aeronautical Information Publication (AIP). The flight crews offered communications support to the Sydney TCU controllers by relaying controller instructions and advice of the power outage to other ATS units.

Maintenance services electrical personnel

All Sydney electrical technical officers took part in a training course provided by the UPS manufacturer in Sydney from 15-17 May 2000. The training was conducted on the operational UPS equipment.

Electrical technical officers usually worked in pairs when servicing the UPS equipment. These officers did not undertake any form of team resource management training to define their tasks while working as a team.

There were two electrical technical officers involved in testing the UPS. One was in the middle of his shift and the other had started duty at 0600 and had extended his shift to conduct the works plan that led to the occurrence.

According to Airservices National Technical Certification program (TechCert), both staff held suitable electrical qualifications to conduct the test.

The Airservices TechCert program assesses technical officers on their knowledge of and ability to safely remove and restore equipment from the national airways system. The TechCert program also required staff to work on the systems within predetermined timeframes for their TechCert certification to be current. Without this currency, staff cannot remove or restore equipment from the national airways system.

Contingency plans

The Sydney Contingency Plan was not activated because of the short duration of the power outage.

The Sydney contingency plan did not include any reference to the loss of the Directed Traffic Information service.

The contingency plan includes reference to radar fail procedures and directs further reference to the degraded mode handbook for which there was no radar fail procedure. It was noted that this document was under review at the time of the incident.

The instructor and student pilot were conducting circuit training. The instructor reported the aircraft was on late downwind when there was a sudden loud bang and the aircraft began to roll to the left. He immediately took control of the aircraft and requested a priority landing. As the aircraft was descending he noticed that all engine indications were normal, however the aircraft required excessive right aileron and right rudder control. A visual inspection revealed the entire right aileron had separated from the aircraft. The descent was continued at a higher than normal speed and the aircraft landed safely.

The separated section of aileron was not recovered. Examination of the remaining aileron to wing attachment (wooden strut) fittings, shows all failures were because of overload. There was no evidence of wood rot or previous damage to any of the fittings.

The ailerons are actuated by a steel control tube from the cockpit, which fits into an aluminium tube (spar) aileron fitting. There is a small shim spacer collar between the inner and outer tubes. The tubes are secured by aluminium pop rivets with steel mandrels.

On the incident aircraft, the inner section of the steel tube shows extensive areas of corrosion (rust), particularly around the securing rivets. There does not appear to be any corrosion proof material present between the tubes or spacer surfaces.

A small broken piece of the aileron (aluminium tube) fitting was still attached to the steel control tube from the cockpit. Part of the broken surface indicates a dark area similar to a pre-existing stress crack, which emanates from a rivet hole. There is also evidence of exfoliation corrosion on this section. The evidence indicates the aileron failure set up at this point. It is probable the aileron then deflected and the air loads tore it from the wing attachments. The initial investigation was able to examine 3 other similar aircraft. Of these, 2 were found to have a significant degree of rust on the inner steel control tube and extensive exfoliation corrosion around the rivet section of the outer aluminium tube.

The SAAB SF 340's departure from Wagga airport had been delayed for 4.5 hours due to ground fog. As the aircraft taxied for departure, the crew completed the pre-flight checks. Incorporated within these checks was the requirement for a "first flight of day" propeller governor overspeed test. As it was the aircraft's first flight for the day, the check was carried out.

Following take-off, and shortly after landing gear retraction, with the Constant Torque On Take-off system engaged, the crew noted that the right engine propeller RPM was low; at approximately 1,100 RPM. The left propeller was within the normal operating range at an indicated 1,378 RPM.

As a return to Wagga was unavailable, due to ground fog, the crew contacted air traffic control indicating their intention to divert to Albury Airport. An Alert Phase was declared and Albury Emergency Services were on stand-by for the aircraft's arrival.

The crew then carried out the appropriate abnormal checklist actions for a propeller underspeed, shutting down the right engine just prior to the top of descent. During that time, the crew briefed the cabin attendant on the engine problem, before informing the passengers of the situation via the aircraft's public address system. Following an uneventful single engine approach and landing, the Alert Phase was cancelled.

An investigation by the aircraft's operator, included analysis of the aircraft's flight data recorder readout. The analysis indicated that during taxi the right propeller RPM had reduced from 1,040 to 990. That RPM drop was consistent with the crew carrying out the propeller governor overspeed test. However, unlike the left propeller, the right propeller RPM had not fully recovered at the completion of the check.

Both crew members reported that on completion of the propeller overspeed governor checks, once they had observed the propeller indications returning towards normal, their attention was diverted towards other checks. The crew also indicated that during the take-off they did not normally check the propeller RPM indications, instead monitoring the engine parameters of "torque and inter-turbine temperature". Consequently, the low right-propeller RPM had not been initially detected.

During the take-off roll the crew noted that the right engine torque had lagged behind the left. That was considered to be due to not having the right power lever pushed far enough forward, as the Constant Torque On Take-off system only engages after advancing the power lever past 64 degrees. The crew had then pushed the right power lever further forward in order to equalise both engine torque indications.

After the flight, the operator completed a thorough maintenance check of the right engine and propeller systems. No unserviceabilities were found during that check. Following consultation with the aircraft's engine manufacturer the aircraft was returned to service. The problem did not re-occur.

The operator attempted to replicate the problem in their SAAB 340 flight simulator. That attempt was observed by a representative from the Civil Aviation Safety Authority. After thorough investigation, the operator was unable to repeat the occurrence.