Assurance-based Learning-enabled Cyber-Physical Systems

Recent advances in machine learning led to the appearance of Learning-Enabled Components (LECs) in Cyber-Physical Systems. LECs are being evaluated and used for various, complex functions including perception and control. However, very little tool support is available for design automation in such systems. This paper introduces an integrated toolchain that supports the architectural modeling of CPS with LECs, but also has extensive support for the engineering and integration of LECs, including support for training data collection, LEC training, LEC evaluation and verification, and system software deployment. Additionally, the toolsuite supports the modeling and analysis of safety cases – a critical part of the engineering process for mission and safety critical systems.

Learning enabled components (LECs) trained using data-driven algorithms are increasingly being used in autonomous robots commonly found in factories, hospitals, and educational laboratories. However, these LECs do not provide any safety guarantees, and testing them is challenging. In this paper, we introduce a framework that performs weighted simplex strategy based supervised safety control, resource management and confidence estimation of autonomous robots. Specifically, we describe two weighted simplex strategies: (a) simple weighted simplex strategy (SW-Simplex) that computes a weighted controller output by comparing the decisions between a safety supervisor and an LEC, and (b) a context-sensitive weighted simplex strategy (CSW-Simplex) that computes a context-aware weighted controller output. We use reinforcement learning to learn the contextual weights. We also introduce a system monitor that uses the current state information and a Bayesian network model learned from past data to estimate the probability of the robotic system staying in the safe working region. To aid resource constrained robots in performing complex computations of these weighted simplex strategies, we describe a resource manager
...

Cyber-Physical Systems (CPS) are used in many applications where they must perform complex tasks with a high degree of autonomy in uncertain environments. Traditional design flows based on domain knowledge and analytical models are often impractical for tasks such as perception, planning in uncertain environments, control with ill-defined objectives, etc. Machine learning based techniques have demonstrated good performance for such difficult tasks, lead- ing to the introduction of Learning-Enabled Components (LEC) in CPS. Model based design techniques have been successful in the development of traditional CPS, and toolchains which apply these techniques to CPS with LECs are being actively developed. As LECs are critically dependent on training and data, one of the key challenges is to build design automation for them. In this pa- per, we examine the development of an autonomous Unmanned Underwater Vehicle (UUV) using the Assurance-based Learning- enabled Cyber-physical systems (ALC) Toolchain. Each stage of the development cycle is described including architectural modeling, data collection, LEC training, LEC evaluation and verification, and system-level assurance.

Machine learning components such as deep neural networks are used extensively in Cyber-physical Systems (CPS). However, they may introduce new types of hazards that can have disastrous consequences and need to be addressed for engineering trustworthy systems. Although deep neural networks offer advanced capabilities, they must be complemented by engineering methods and practices that allow effective integration in CPS. In this paper, we investigate how to use the conformal prediction framework for assurance monitoring of CPS with machine learning components. In order to handle high-dimensional inputs in real-time, we compute nonconformity scores using embedding representations of the learned models. By leveraging conformal prediction the approach provides well-calibrated confidence and can allow monitoring that ensures a bounded small error rate while limiting the number of inputs for which an accurate prediction cannot be made. Empirical evaluation results using the German Traffic Sign Recognition Benchmark and a robot navigation dataset demonstrate that the error rates are well-calibrated while the number of alarms is small. The method is computationally efficient, and therefore, the approach is promising for assurance monitoring of CPS.

Cyber Physical Systems (CPS) have increasingly started using Learning Enabled Components (LECs) for performing perception based control tasks. The simple design approach, and their capability to continuously learn has led to their widespread use in different autonomous applications. Despite their simplicity and impressive capabilities, these models are difficult to assure, which makes their use challenging. The problem of assuring CPS with controllers has been achieved using the Simplex Architecture. This architecture integrates the system to be assured with a safe controller and provides a decision logic to switch between the decisions of these controllers. However, the key challenges in using the Simplex Architecture are: (1) designing an effective decision logic, and (2) sudden transitions between controller decisions lead to inconsistent system performance. To address these research challenges, we make three key contributions: (1) dynamic-weighted simplex strategy – we introduce “weighted simplex strategy" as the weighted ensemble extension of the classical Simplex Architecture. We then provide a reinforcement learning based mechanism to find dynamic ensemble weights, (2) middleware framework – we design a framework that allows the use of the dynamic-weighted simplex strategy, and provides a resource manager to monitor the computational resources, and (3) hardware – we design a remote-controlled car called DeepNNCar to test and demonstrate the aforementioned key concepts. Using the hardware, we show that the dynamic-weighted simplex strategy has 60% fewer out-of-track occurrences (soft constraint violations), while demonstrating higher optimized speed (performance) of 0.4 m/s during indoor driving than the original LEC driven system.

Safety Case has become an integral component for safety-certification in various Cyber Physical System domains including automotive, aviation, medical devices, and military. The certification processes for these systems are stringent and require robust safety assurance arguments and substantial evidence backing. Despite the strict requirements, current practices still rely on manual methods that are brittle, do not have a systematic approach or thorough consideration of sound arguments. In addition, stringent certification requirements and ever-increasing system complexity make ad-hoc, manual assurance case generation (ACG) inefficient, time consuming, and expensive. To improve the current state of practice, we introduce a structured ACG tool which uses system design artifacts, accumulated evidence, and developer expertise to construct a safety case and evaluate it in an automated manner. We also illustrate the applicability of the ACG tool on a remote-control car testbed case study.

This demo showcases the features of an adaptive middleware framework for resource constrained autonomous robots like DeepNNCar (Figure 1). These robots use Learning Enabled Components (LECs), trained with deep learning models to perform control actions. However, these LECs do not provide any safety guarantees and testing them is challenging. To overcome these challenges, we have developed an adaptive middleware framework that (1) augments the LEC with safety controllers that can use different weighted simplex strategies to improve the systems safety guarantees, and (2) includes a resource manager to monitor the resource parameters (temperature, CPU Utilization), and offload tasks at runtime. Using DeepNNCar we will demonstrate the framework and its capability to adaptively switch between the controllers and strategies based on its safety and speed performance.

This manuscript evaluates the safety of a neural network controller that seeks to ensure that an Unmanned Underwater Vehicle (UUV) does not collide with a static object in its path. To achieve this, we utilize methods that can determine the exact output reachable set of all the UUV’s components through the use of star-sets. The star-set is a computationally efficient set representation adept at characterizing large input spaces. It supports cheap and efficient computation of affine mapping operations and intersections with half-spaces. The system under consideration in this work represents a more complex system than Neural Network Control Systems (NNCS) previously consideredin other works, and consists of a total of four components. Our experimental evaluation uses four different scenarios to show that our star-set based methods are scalable and can be efficiently used to analyze the safety of real-world cyber-physical systems (CPS).

Learning Enabled Components (LECs) are widely being used in a variety of perceptions based autonomy tasks like image segmentation, object detection, end-to-end driving, etc. These components are trained with large image datasets with multimodal factors like weather conditions, time-of-day, traffic-density, etc. The LECs learn from these factors during training, and while testing if there is variation in any of these factors, the components get confused resulting in low confidence predictions. Those images with factor values, not seen, during training are commonly referred to as Out-of-Distribution (OOD). For safe autonomy, it is important to identify the OOD images, so that a suitable mitigation strategy can be performed. Classical one-class classifiers like SVM and SVDD are used to perform OOD detection. However, multiple labels attached to images in these datasets restrict the direct application of these techniques. We address this problem using the latent space of the $\beta$-Variational Autoencoder ($\beta$-VAE). We use the fact that compact latent space generated by an appropriately selected $\beta$-VAE will encode the information about these factors in a few latent variables, and that can be used for quick and computationally inexpensive detection. We evaluate our approach on the dataset, and our results show the latent space of $\beta$-VAE is sensitive to encode changes in the values of the generative factor. 
...

Development of Cyber Physical Systems (CPSs) requires close interaction between developers with expertise in many domains to achieve ever-increasing demands for improved performance, reduced cost, and more system autonomy. Each engineering discipline commonly relies on domain-specific modeling languages, and analysis and execution of these models is often automated with appropriate tooling. However, integration between these heterogeneous models and tools is often lacking, and most of the burden for inter-operation of these tools is placed on system developers. To address this problem, we introduce a workflow modeling language for the automation of complex CPS development processes and implement a platform for execution of these models in the Assurance-based Learning-enabled CPS (ALC) Toolchain. Several illustrative examples are provided which show how these workflow models are able to automate many time-consuming integration tasks previously performed manually by system developers.

Hoang-Dung Tran, Xiaodong Yang, Diego Manzanas Lopez, Patrick Musau, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, Taylor T. Johnson, "NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems", In 32nd International Conference on Computer-Aided Verification (CAV), 2020, July. 

Stanley Bak, Hoang-Dung Tran, Kerianne Hobbs, Taylor T. Johnson, "Improved Geometric Path Enumeration for Verifying ReLU Neural Networks", In 32nd International Conference on Computer-Aided Verification (CAV), 2020, July.

Hoang-Dung Tran, Stanley Bak, Weiming Xiang, Taylor T. Johnson, "Verification of Deep Convolutional Neural Networks Using ImageStars", In 32nd International Conference on Computer-Aided Verification (CAV), Springer, 2020, July.

Cyber-physical systems (CPS) can benefit by the use of learning enabled components (LECs) such as deep neural networks (DNNs) for perception and decision making tasks. However, DNNs are typically non-transparent making reasoning about their predictions very difficult, and hence their application to safety-critical systems is very challenging. LECs could be integrated easier into CPS if their predictions could be complemented with a confidence measure that quantifies how much we trust their output. The paper presents an approach for computing confidence bounds based on Inductive Conformal Prediction (ICP). We train a Triplet Network architecture to learn representations of the input data that can be used to estimate the similarity between test examples and examples in the training data set. Then, these representations are used to estimate the confidence of set predictions from a classifier that is based on the neural network architecture used in the triplet. The approach is evaluated using a robotic navigation benchmark and the results show that we can computed trusted confidence bounds efficiently in real-time.

Perception-based deep neural networks used in Cyber Physical Systems are known to fail when faced with inputs that are out-of-distribution (OOD). OOD detection is a complex problem as we need to first identify the shift in the test data from the training distribution and then we need to isolate the responsible generative factor(s) (weather, lighting levels, traffic density, etc.). Unlike the state of the art that uses multi-chained one-class classifiers, we propose an efficient single monitor that uses the principle of disentanglement to train the latent space of a variational autoencoder to be sensitive to distribution shifts in different generative factors. We demonstrate our approach using an end-to-end driving controller in the CARLA simulator.  
...

Autonomous Cyber Physical Systems (CPSs) are often required to handle uncertainties and self-manage the system operation in response to problems and increasing risk in the operating paradigm. This risk may arise due to distribution shifts, environmental context, or failure of software or hardware components. Traditional techniques for risk assessment focus on design-time techniques such as hazard analysis, risk reduction, and assurance cases among others. However, these static, design-time techniques do not consider the dynamic contexts and failures the systems face at runtime. We hypothesize that this requires a dynamic assurance approach that computes the likelihood of unsafe conditions or system failures considering the safety requirements, assumptions made at design time, past failures in a given operating context, and the likelihood of system component failures. We introduce the ReSonAte dynamic risk estimation framework for autonomous systems. ReSonAte reasons over Bow-Tie Diagrams (BTDs) which capture information about hazard propagation paths and control strategies. Our innovation is the extension of the BTD formalism with attributes for modeling the conditional relationships with the state of the system and environment. We also describe a technique for estimating these conditional relationships and equations for estimating risk based on the state of the system and environment. To help with this process, we provide a scenario modeling procedure that can use the prior distributions of the scenes and threat conditions to generate the data required for estimating the conditional relationships. To improve scalability and reduce the amount of data required, this process considers each control strategy in isolation and composes several single-variate distributions into one complete multi-variate distribution for the control strategy in question. Lastly, we describe the effectiveness of our approach using two separate autonomous system simulations: CARLA and an unmanned underwater vehicle.

Autonomous systems use extensively learning-enabled components such as deep neural networks (DNNs) for prediction and decision making. In this paper, we utilize a feedback loop between learning-enabled components used for classification and the sensors of an autonomous system in order to improve the confidence of the predictions. We design a classifier using Inductive Conformal Prediction (ICP) based on a triplet network architecture in order to learn representations that can be used to quantify the similarity between test and training examples. The method allows computing confident set predictions with an error rate predefined using a selected significance level. A feedback loop that queries the sensors for a new input is used to further refine the predictions and increase the classification accuracy. The method is computationally efficient, scalable to high-dimensional inputs, and can be executed in a feedback loop with the system in real-time. The approach is evaluated using a traffic sign recognition dataset and the results show that the error rate is reduced.

Hazard analysis and assurance cases are well-established approaches for assessing the safety of cyber-physical systems and it is already being extended to address the impact of the use of learning enabled controllers in the system. However, such static assurances cases are not suitable for dynamic situations where the context and the assumptions made while developing the assurance case may be invalid or only partially correct. This requires a dynamic assurance approach, which however is highly non-trivial and needs to consider the composition of various hypotheses about failures in different aspects of the system including the learning enabled component, safety requirements, potential hazard conditions in the operating environment, hazard prevention, and mitigation strategies. This paper introduces ReSonAte which uses the information gathered by hazard analysis and assurance cases to build Bow-Tie Diagrams to model hazard propagation paths and capture their relationships with the state of the system and environment. These Bow-tie diagrams are used to synthesize graphical models that are then used at runtime along with the information gathered from prior incidents about the possible environmental hazards and the hypothesis from failure diagnosers and system runtime monitors to estimate the hazard rates at runtime. These hazard rates are then used to determine the likelihood of unsafe system-level consequences captured in the bow-tie diagram.  We implement ReSonAte for an autonomous vehicle example in the CARLA simulator and through comprehensive simulations across 600 executions we show that there is a strong correlation between our risk estimates and eventual vehicular collisions. Our approach outperforms static risk estimates produced from the assurance cases. Also, the Bow-Tie based risk calculations on an average take 0.3 milliseconds at runtime in addition to the overhead introduced by the assurance monitors and runtime detectors used to identify various anomalies.

Deep Neural Networks (DNNs) are widely used in automotive Cyber-Physical Systems (CPS) to implement autonomy related tasks. However, these networks have exhibited erroneous predictions to anomalous inputs that manifest either due to Out-of-Distribution (OOD) data or adversarial attacks. To detect these anomalies, a separate DNN called assurance monitor is used in parallel to the controller DNN, increasing the resource burden and latency. We hypothesize that a single network that can perform controller predictions and anomaly detection is necessary to reduce the resource requirements. Deep-Radial Basis Function (RBF) networks provide a rejection class alongside the class predictions, which can be used for anomaly detection. However, the use of RBF activation functions limits the applicability of these networks to only classification tasks. In this paper, we discuss the steps involved in detecting anomalies in CPS regression and classification tasks. Further, we design deep-RBF networks using popular DNNs such as NVIDIA DAVE-II and ResNet20 and then use the resulting rejection class for detecting physical and data poison adversarial attacks. We show that the deep-RBF network can effectively detect these attacks with limited resource requirements. 
...

Machine learning components such as deep neural networks are used extensively in Cyber-Physical Systems (CPS). However, such components may introduce new types of hazards that can have disastrous consequences and need to be addressed for engineering trustworthy systems. Although deep neural networks offer advanced capabilities, they must be complemented by engineering methods and practices that allow effective integration in CPS. In this paper, we proposed an approach for assurance monitoring of learning-enabled CPS based on the conformal prediction framework.
...

Cyber-physical systems (CPSs) use learning-enabled components (LECs) extensively to cope with various complex tasks under high-uncertainty environments. However, the dataset shifts between the training and testing phase may lead the LECs to become ineffective to make large-error predictions, and further, compromise the safety of the overall system. In our paper, we first provide the formal definitions for different types of dataset shifts in learning-enabled CPS. Then, we propose an approach to detect the dataset shifts effectively for regression problems.
...

Hoang-Dung Tran, Neelanjana Pal, Patrick Musau, Xiaodong Yang, Nathaniel P. Hamilton, Diego Manzanas Lopez, Stanley Bak, Taylor T. Johnson, "Robustness Verification of Semantic Segmentation Neural Networks using Relaxed Reachability", In 33rd International Conference on Computer-Aided Verification (CAV), Springer, 2021, July. http://www.taylortjohnson.com/research/tran2021cav.pdf 

Weiming Xiang, Hoang-Dung Tran, Xiaodong Yang, Taylor T. Johnson, "Reachable Set Estimation for Neural Network Control Systems: A Simulation-Guided Approach", In IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 5, pp. 1821-1830, 2021, May. http://www.taylortjohnson.com/research/xiang2021tnnls.pdf

Diego Manzanas Lopez, Taylor T. Johnson, Hoang-Dung Tran, Stanley Bak, Xin Chen, Kerianne Hobbs, "Verification of Neural Network Compression of ACAS Xu Lookup Tables with Star Set Reachability", In AIAA Scitech 2021 Forum, AIAA, 2021, January. http://www.taylortjohnson.com/research/lopez2021scitech.pdf

Deep Neural Networks are actively being used in the design of autonomous Cyber-Physical Systems (CPSs). The advantage of these models is their ability to handle high-dimensional state-space and learn compact surrogate representations of the operational state spaces. However, the problem is that the sampled observations used for training the model may never cover the entire state space of the physical environment, and as a result, the system will likely operate in conditions that do not belong to the training distribution. These conditions that do not belong to training distribution are referred to as Out-of-Distribution (OOD). Detecting OOD conditions at runtime is critical for the safety of CPS. In addition, it is also desirable to identify the context or the feature(s) that are the source of OOD to select an appropriate control action to mitigate the consequences that may arise because of the OOD condition. In this paper, we study this problem as a multi-labeled time series OOD detection problem over images, where the OOD is defined both sequentially across short time windows (change points) as well as across the training data distribution. A common approach to solving this problem is the use of multi-chained one-class classifiers. However, this approach is expensive for CPSs that have limited computational resources and require short inference times. Our contribution is an approach to design and train a single $\beta$-Variational Autoencoder detector with a partially disentangled latent space sensitive to variations in image features. We use the feature sensitive latent variables in the latent space to detect OOD images and identify the most likely feature(s) responsible for the OOD. We demonstrate our approach using an Autonomous Vehicle in the CARLA simulator and a real-world automotive dataset called nuImages.

Autonomous Cyber-Physical Systems (CPS) must be robust against potential failure modes, including physical degradations and software issues, and are required to self-manage contingency actions for these failures. Physical degradations often have a significant impact on the vehicle dynamics causing irregular behavior that can jeopardize system safety and mission objectives. The paper presents a novel Behavior Tree-based autonomy architecture that includes a Fault Detection and Isolation Learning-Enabled Component (FDI LEC) with an Assurance Monitor (AM) designed based on Inductive Conformal Prediction (ICP) techniques. The architecture implements real-time contingency-management functions using fault detection, isolation and reconfiguration subsystems. To improve scalability and reduce the false-positive rate of the FDI LEC, the decision-making logic provides adjustable thresholds for the desired fault coverage and acceptable risk. The paper presents the system architecture with the integrated FDI LEC, as well as the data collection and training approach for the LEC and the AM. Lastly, we demonstrate the effectiveness of the proposed architecture using a simulated autonomous underwater vehicle (AUV) based on the BlueROV2 platform

This video presents a step-by-step example how to work with the IDE in the ALC Toolchain running cp1_02 and cp1_03 scenarios (low battery and return to home failsafe):

• Starting IDE using IDELaunch plugin
• Overview of a launch script
...

This is an example how to use the ALC Toolchain and run example scenarios in headless mode. Also presents how to plot the results after a finished simulation.

• Go to ALC > 2. Construction > Testing
• Select a scenario to launch ...

Behaviour Tree complex example with RQT for behaviour trees, RViz and PlotJuggler in the ALC Toolchain

• This video presents how to use the IDE in ALC Toolchain to work and visualize Behaviour Trees
• Run IDELaunch plugin to start IDE first
...

This simulation presents a pipe tracking scenario, default parameters and high battery level with static box obstacles.
With use_behaviour_tree = true, the autonomy is controlled by a BT based logic in the ALC Toolchain.
Obstacle avoidance is enabled by default during the whole mission, and obstacles will spawn directly ahead of the UUV. UUV applies obstacle avoidance and after maneuver it returns to Pipe Following.
...

CP1_01: Geofence RTH

This simulation presents a pipe tracking scenario with Geofence failsafe and Return To Home (RTH).
With use_behaviour_tree = true, the autonomy is controlled by a BT based logic in the ALC Toolchain.
...

These videos are presenting a CP2 Thruster Degradation scenario with Fault Detection, Isolation and control Reallocation (CP2_degradation).
'cp2_with_ide_in_alc' video presents the simulation scenario running from the toolchain. The same is presented in 'cp2_in_detail', but with explanation and using PlotJuggler also.
...

This simulation presents a waypoint following scenario, with a Flight Data Recorder spawned inside the waypoints covered area (marked with red square). UUV completes mission, while gathering FDR receiver data. After the mission is completed, UUV estimates FDR position, spawns a new waypoint (white sphere), goes there and loiters around it. An obstacle is spawned ahead the UUV, what is interfering with the search pattern and the loiter path also.
...

This simulation presents a waypoint following scenario where the waypoints are randomly generated. Cross track error minimization can be enabled or disabled for this scenario in the ALC Toolchain.

This simulation presents a waypoint following scenario, with a wide "wall" obstacle. UUV avoids the obstacle and returns to the path immediately. Cross Track Error (x_track_error=true) is minimized for that.
With use_behaviour_tree = true, the autonomy is controlled by a BT based logic in the ALC Toolchain.

This simulation presents a waypoint following scenario, where a static obstacle is detected close to waypoint #3. UUV alters the waypoint - moves it forward towards to the next waypoint.
When it is closer to a given threshold to the next waypoint, UUV skips the already altered one.
...

This video presents the VU FLS LEC (LSTM noise filtering LEC) with VAE Assurance Monitor.
This LEC is using the FLS pencilbeam range measurement (with extra noise applied)
The LEC and AM was trained on small amount of sim data.
...

This video presents a real-time hazard rate calculation​ with ReSonAte framework based on a CP2 thruster degradation scenario​.
ReSonAte is using onboard LEC and AM data only as input - no ground truth data was used.

• Degradation starts at t=50s ​
  ...

Several orders of magnitude progress made analyzing learning-enabled components (LECs) like neural networks and usage in autonomous CPS at design-time with NNV and other approaches (scalability, layer types, closed-loop interaction, etc.)
However, while improving confidence of such LECs before they are deployed is important, online monitoring at runtime is essential
...