Regulatory surveillance – scoping of surveillance events

Overview

CASA acknowledges the recommendation and agrees that formal procedures and guidance in relation to scoping the nature of the operator’s approved activities is important to identify the threats and hazards associated with those activities and provide essential information as to how the operator manages their operational risk.

CASA’s surveillance methodology utilises risk-based processes to inform oversight of an authorisation holder’s regulatory compliance and safety performance. This structured process is used to prioritise surveillance activities and inform audit scope decision making. Surveillance is focused on the Authorisation Holder’s regulatory compliance and operational effectiveness in managing their risks and is also a method by which CASA can validate that activities conducted by the Authorisation Holder are being conducted safely.

The way in which surveillance is prioritised and scoped is facilitated through the on-going interactions and formalised reviews of the operator, called Authorisation Holder Assessments (AHA’s). As part of the ongoing oversight of Authorisation Holders, Authorisation Management Teams (AMT’s) follow a structured assessment process to identify possible emerging risk relating to the operations of the Authorisation Holder. Surveillance activities are subsequently planned to enable effective validation of relevant systems and elements.

The AHA is an assessment of the apparent risk to safety presented by an authorisation holder. The AHA provides a consolidation of information to assist an authorisation management team to determine the surveillance priority of an authorisation holder. It is completed by considering the following information, most of which is presented in Sky Sentinel:

• outstanding Safety Findings and findings history
• date of the last Authorisation Holder Performance Indicator (AHPI) assessment and AHPI assessment history
• AHPI result category
• the number of AHPI sub-factors recorded as ‘Don’t know’
• time since the last Level 1 and Level 2 surveillance event, particularly when compared to the recommended frequency specified in the currency guide
• any additional surveillance intelligence about the authorisation holder and their operation.

On completion of the assessment process, the authorisation management team makes recommendations for surveillance and safety actions in preparation for the monthly meeting of the Surveillance Priority Review Group (SPRG). These recommendations are recorded in Sky Sentinel as surveillance requests.

The assessment process is at the heart of the authorisation management team system. It is an ongoing process that occurs through the management/oversight of an authorisation holder or like groups of authorisation holders. The AHA provides a consolidation of information to assist an authorisation management team to determine the surveillance priority of an authorisation holder.

Additionally, in 2019 CASA instigated a “Surveillance Refresher” training program that ensures that all inspectors carrying out surveillance events are fully cognoscente of CSM requirements. To date 97% of the inspectorate and all Surveillance Technical Officers have undergone this training.

Current State

CASA currently provides guidance for its policy and procedures in relation to the conduct of surveillance within the CASA Surveillance Manual (CSM) (v.4.2 – July 2019).

This manual reflects surveillance management concepts and processes that allow for the prioritisation of surveillance activities based on potential risk and to determine what areas of a system should be addressed in a surveillance event. The manual sets out the processes to be followed when conducting surveillance on civil aviation authorisation holders and contains information regarding surveillance scoping in its various sections.

Currently the CSM references ‘SCOPING’ at Section 2.8 - Surveillance scheduling;

• CASA’s surveillance program scheduling is driven by the risk to safety posed by authorisation holders and is based on an assessment of a number of factors. These factors include the assessment of an authorisation holder’s safety performance, taking into account assessment factors indicated by the Authorisation Holder Performance Indicator (AHPI) assessment results and time since the last assessment, outstanding Safety Findings and findings history, time since the last surveillance event, and safety-related risks specific to each authorisation holder. Based on this consolidated information, CASA has the ability to prioritise surveillance activities commensurate with resources available.

At Section 3.2.1 of the CSM – Authorisation Holder Assessment (Assess)

• The purpose of this process is for the authorisation management team to assess all available information relating to an authorisation holder’s activities. This assessment allows for the identification of areas of concern and the development of proposals for surveillance to be considered in the surveillance priority review process. The output from this process step is the surveillance request.
• In July 2019 the CSM was amended (Section 4.4 Surveillance event preparation) to include a detailed description of the Surveillance Technical Officer (STO) role which plays an important part in the planning and scoping of audits. In addition to the STO role the amendment provided more detailed description on event preparation and removed the optional use of all of the surveillance event preparation forms and has made the use of these forms’ mandatory.

At section 4.4.4.2 Process Details - Prepare for Level 1 surveillance event

• Note: If a Level 1 surveillance event is to be proposed, the scope of the proposed event must be defined in Sky Sentinel by selecting the appropriate systems and elements to be covered. A surveillance request may be scoped to the system risk level if considered appropriate.
• Note: Category 1 NSSP events will have several prepopulated scope items.

The Sky Sentinel Surveillance Scoping Aid is used as a reference to show the system risks for which the effectiveness of an authorisation holder’s control has been assessed. Additionally, taking into consideration the size and complexity of an individual authorisation holder’s operation, all systems and elements must be assessed in a timely manner.

To develop a Surveillance Worksheet, the inspector will need to review a number of documents such as the authorisation holder’s systems risk history (Sky Sentinel), organisational policy and procedures manuals and identify specific areas and risks to be assessed or reviewed as identified in the Surveillance Checklist. The scope and depth of each surveillance event will vary depending on the information, data and history known about the authorisation holder. See Surveillance Planning and Scoping Form (Form 1189) or other scoping forms.

Future State

Currently the surveillance and scoping form (Form 1189) is being amended to add the review of ‘current activities’ and will be incorporated in the next CSM amendment (Due Approx. Jan 2020). The use of Form 1189 for scoping of surveillance events became mandatory in July 2019.

Additionally, in December 2019 CASA’s Safety Intelligence and Analysis Section developed an “Operator Profile Report” using the Power BI tool that is being utilised by Surveillance Technical Officers (STO’s) and Auditors in event preparation. The use of this tool will be included in the next CSM amendment.

Summary

CASA has recently mandated use of the Surveillance and Scoping Form (Form 1189) which provides guidance on the detail, information, data, known history about the authorisation holder and scoped system elements.

The current Form 1189 is being amended to require consideration of ‘current activities’ and will be incorporated in the next CSM amendment (Due Approx. Jan 2020).

The proposed addition of an Operator Profile Report, which provides contextual data on the current state of an operator, with historical information related to past surveillance activity.

CASA is of the view that these recent amendments to the CSM, including the proposed amended scoping form (due for implementation by end January 2020) provide a more effective audit scoping process in which consideration and documentation of an operator’s activities is mandatory.

ATSB comment: 21 January 2020
The ATSB noted the CASA response and requested some additional information before assessing the response.

CASA response: 4 March 2020
CASA provided the ATSB a copy of the new Surveillance planning and scoping development form and an example of the new Operator risk profile.

ATSB response: 5 March 2020
The ATSB welcomes CASA’s detailed response to this recommendation. The ATSB notes that the Surveillance planning and scoping development form is now mandatory for planning surveillance events, and the new version of the form includes many additional fields compared to previous versions of this form, including the organisation’s current activities, current permissions and approvals, industry intelligence, third-party audits and the organisation’s manuals. The ATSB also notes that the new operator risk profile prepared prior to a surveillance event includes numerous items of information about the organisation being audited in one document.