Neural networks can obtain state-of-the-art performance in various tasks, including image classification, object detection, speech recognition, and machine translation. Due to this impressive performance, there has been a desire to utilize neural networks for applications in industries with safety-critical components, such as aerospace, automotive, and healthcare. However, while these industries have established processes for verifying and validating traditional software, it is often unclear how to verify the reliability of neural networks. This issue is especially prevalent in aviation, where there is potential to revolutionize the industry. However, existing airborne certification standards present major incompatibilities with Machine Learning technology. These include issues with ML model traceability and explainability and the inadequacy of traditional coverage metrics. The certification of ML-based airborne systems is problematic due to these incompatibilities. Furthermore, new certification standards intended to address these challenges are not yet released.

In this talk, we’ll introduce a case study for certifying an airborne machine learning system. We’ll build a runway sign classification system that receives images from a forward-facing camera in the aircraft and then detects airport runway signs, aiding the pilot in navigation and situational awareness at the airport. We propose and implement a custom ML certification workflow for machine learning systems based on existing certification standards to tackle the previously mentioned challenges. We will walk you through all the steps in the workflow, from defining the ML requirements, managing the data, training the model, and verifying its performance to the implementation of the system in hardware and validation of the requirements. This case study will provide insights and potential solutions across industries with safety-critical components seeking to integrate neural networks into their operations.