Home » All posts

The Implicit Controllability Pitfall for autonomous system safety

Posted by badrut Monday, February 11, 2019 0 comments
The Implicit Controllability Pitfall:
A safety case must account for not only failures within the autonomy system, but also failures within the vehicle. With a fully autonomous vehicle, responsibility for managing potentially unsafe equipment malfunctions that were previously mitigated by a human driver falls to the autonomy.


A subtle pitfall when arguing based on conformance to a safety standard is neglecting that the assumptions made when assessing the subsystem that have potentially been violated or changed by the use of autonomy. Of particular concern for ground vehicles is the “controllability” aspect of an ISO 26262 ASIL analysis. (Severity and exposure might also change for an autonomous vehicle due to different usage patterns and should also be considered, but are beyond the scope of this discussion.)

The risk analysis of an underlying conventional vehicle according to ISO 26262 requires taking into account the severity, exposure, and controllability of each hazard (ISO 2011). The controllability aspect assumes a competent human driver is available to react to and mitigate equipment malfunctions. Taking credit for some degree of controllability generally reduces the integrity requirements of a component. This idea is especially relevant for Advanced Driver-Assistance Systems (ADAS) safety arguments, in which it is assumed that the driver will intervene in a timely manner to correct any vehicle misbehavior, including potential ADAS malfunctions.

With a fully autonomous vehicle, responsibility for managing potentially unsafe equipment malfunctions that were previously mitigated by a human driver falls to the autonomy. That means that all the assumed capabilities of a human driver that have been built in to the safety arguments regarding under-lying vehicle malfunctions are now imposed upon the autonomy system.

If the autonomy design team does not have access to the analysis behind underlying equipment safety arguments, there might be no practical way to know what all the controllability assumptions are. In other words, the makers of an autonomy kit might be left guessing what failure response capabilities they must provide to preserve the correctness of the safety argumentation for the underlying vehicle.

The need to mitigate some malfunctions is likely obvious, but we have found that “obvious” is in the eye of the beholder. Some examples of assumed human driver interventions we have noted, or even experienced first-hand include:
  • Pressing hard on the brake pedal to compensate for loss of power assist 
  • Controlling the vehicle in the event of a tire blowout 
  • Path planning after catastrophic windshield damage from debris impact 
  • Manually pumping brakes when anti-lock brake mechanisms time out due to excessive activation on slick surfaces
  • Navigating by ambient light starlight after a lighting system electrical failure at speed while bring vehicle to a stop
  • Attempting to mitigate the effects of uncommanded propulsion power 
To the degree that the integrity level requirements of the vehicle were reduced by the assumption that a human driver could mitigate the risk inherent in such malfunction scenarios, the safety case is insufficient if an autonomy kit does not have comparable capabilities.

Creating a thorough safety argument will require either obtaining or re-verse engineering all the controllability assumptions made in the design of the underlying vehicle. Then, the autonomy must be assessed to have an adequate ability to provide the safety relevant controllability assumed in the vehicle design, or an alternate safety argument must be made.

For cases in which the controllability assumptions are not available, there are at least two approaches that should both be used by a prudent design team. First,  FMEA, HAZOP, and other appropriate analyses should be performed on vehicle components and safety relevant functions to ensure that the autonomy can react in a safe way to malfunctions. Such an analysis will likely struggle with whether or not it is safe to assume that the worst types of malfunctions will be adequately mitigated by the vehicle without autonomy intervention.

Second, defects reported on comparable production vehicles should be considered as credible malfunctions of the non-autonomous portions of any vehicle control system since they have already happened in production systems. Such malfunctions include issues such as the drivetrain reporting the opposite of the current direction of motion, uncommanded acceleration, significant braking lag, loss of headlights, and so on (Koopman 2018a).

(This is an excerpt of our SSS 2019 paper:  Koopman, P., Kane, A. & Black, J., "Credible Autonomy Safety Argumentation," Safety-Critical Systems Symposium, Bristol UK, Feb. 2019.  Read the full text here)
  • ISO (2011) Road vehicles -- Functional Safety -- Management of functional safety, ISO 26262, International Standards Organization, 2011.
  • Koopman, P., (2018a), Potentially deadly automotive software defects, https://betterembsw.blogspot.com/2018/09/potentially-deadly-automotive-software.html, Sept. 25, 2018.

Edge Cases and Autonomous Vehicle Safety -- SSS 2019 Keynote

Posted by badrut Wednesday, February 6, 2019 0 comments
Here is my keynote talk for SSS 2019 in Bristol UK.

Edge Cases and Autonomous Vehicle Safety

Making self-driving cars safe will require a combination of techniques. ISO 26262 and the draft SOTIF standards will help with vehicle control and trajectory stages of the autonomy pipeline. Planning might be made safe using a doer/checker architectural pattern that uses  deterministic safety envelope enforcement of non-deterministic planning algorithms. Machine-learning based perception validation will be more problematic. We discuss the issue of perception edge cases, including the potentially heavy-tail distribution of object types and brittleness to slight variations in images. Our Hologram tool injects modest amounts of noise to cause perception failures, identifying brittle aspects of perception algorithms. More importantly, in practice it is able to identify context-dependent perception failures (e.g., false negatives) in unlabeled video.



Command Override Anti-Pattern for autonomous system safety

Posted by badrut Monday, February 4, 2019 0 comments
Command Override Anti-Pattern:
Don't let a non-critical "Doer" over-ride the safety critical "Checker."  If you do this, the Doer can tell the Checker that something is safe when it isn't.


https://pixabay.com/en/base-jump-jump-base-jumper-leaping-1600668/

(First in a series of postings on pitfalls and fallacies we've seen used in safety assurance arguments for autonomous vehicles.)

A common pitfall when identifying safety relevant portions of a system is overlooking the safety relevance of sensors, actuators, software, or some other portion of a system. A common example is creating a system that permits the Doer to perform a command override of the Checker. (In other words, the designers think they are building a Doer/Checker pattern in which only the Checker is safety relevant, but in fact the Doer is safety relevant due to its ability to override the Checker’s functionality.)

The usual claim being made is that a safety layer will prevent any malfunction of an autonomy layer from creating a mishap. This claim generally involves arguing that an autonomy failure will activate a safing function in the safety layer, and that an attempt by the autonomy to do something unsafe will be prevented. A typical (deficient) scheme is to have autonomy failure detected via some sort of self-test combined with the safety layer monitoring an autonomy layer heartbeat signal. It is further argued that the safety layer is designed in conformance with a suitable functional safety standard, and therefore acts as a safety-rated Checker as part of a Doer/Checker pair.

The flaw in that safety argumentation approach is that the autonomy layer has been presumed to fail silent via a combination of self-diagnosed fault detection and lack of heartbeat. However, self-diagnosis and heartbeat detection methods provide only partial fault detection (Hammett 2001). For example, there is always the possibility that the checking function itself has been com-promised by a fault that leads to false negatives of checks for incorrect functionality.

As a simple example, a heartbeat signal might be generated by a timer interrupt in the autonomy computer that continues to function even if significant portions of the autonomy software have crashed or are generating incorrect results. In general, such an architectural pattern is unsafe because it permits a non-silent failure in the autonomy layer to issue an unsafe vehicle trajectory command that overrides the safety layer. Fixing this fault requires making the autonomy layer safety-critical, which defeats a primary purpose of using a Doer/Checker architecture.

In practice, safety layer logic is usually less permissive than the autonomy layer. By less permissive, we mean that it under-approximates the safe state space of the system in exchange for simplifying computation (Machin et al. 2018). As a practical example, the safety layer might leave a larger spatial buffer area around obstacles to simplify computations, resulting in a longer total path length for the vehicle or even denying the vehicle entry into situations such as a tight alleyway that is only slightly larger than the vehicle.

A significant safety compromise can occur when vehicle designers attempt to increase permissiveness by enabling a non-safety-rated autonomy layer to say “trust me, this is OK” to override the safety layer. This creates a way for a malfunctioning autonomy layer to override the safety layer, again subverting the safety of the Doer/Checker pair architecture.
Eliminating this command override anti-pattern requires that the designers accept that there is an inherent trade-off between permissiveness and simplicity. A simple Checker tends to have limited permissiveness. Increasing permissiveness makes the Checker more complex, increasing the fraction of the system design work that must be done with high integrity. Permitting a lower integrity Doer to override the safety-relevant behavior of a high integrity Checker in an attempt to avoid Checker complexity is unsafe.

Related pitfalls are first a system in which the Checker only covers a sub-set of the safety properties of the system. This implicitly trusts the Doer to not have certain classes of defects, including potentially requirements defects. If the Checker does not actually check some aspects of safety, then the Doer is in fact safety relevant. A second pitfall is having the Checker supervise a diagnosis operation for a Doer health check. Even if the Doer passes a health check, that does not mean its calculations are correct. At best it means that the Doer is operating as designed – which might be unsafe since the Doer has not been verified with the level of rigor required to assure safety.

We have found it productive to conduct the following thought experiment when evaluating Doer/Checker architectural patterns and other systems that rely upon assuring the integrity of only a subset of safety-related functions. Ask this question: “Assume the Doer (a portion of the system, including software, sensors, and actuators, that are not ‘safety related’) maliciously at-tempts to compromise safety in the worst way possible, with full and complete knowledge of every aspect of the design. Could it compromise safety?” If the answer is yes, then the Doer is in fact safety relevant, and must be designed to a sufficiently high level of integrity. Attempts to argue that such an outcome is unlikely in practice must be supported by strong evidence.

(This is an excerpt of our SSS 2019 paper:  Koopman, P., Kane, A. & Black, J., "Credible Autonomy Safety Argumentation," Safety-Critical Systems Symposium, Bristol UK, Feb. 2019.  Read the full text here)
  • Hammett, R., (2001) “Design by extrapolation: an evaluation of fault-tolerant avion-ics, 20th Conference on Digital Avionics Systems, IEEE, 2001.
  • Machin, M., Guiochet, J., Waeselynck, H., Blanquart, J-P., Roy, M., Masson, L., (2018) “SMOF – a safety monitoring framework for autonomous systems,” IEEE Trans. System, Man and Cybernetics Systems, 48(5) May 2018, pp. 702-715.


Credible Autonomy Safety Argumentation Paper

Posted by badrut Friday, February 1, 2019 0 comments
Here is our paper on pitfalls in safety argumentation for autonomous systems for SSS 2019.  My keynote talk will mostly be about perception stress testing but I'm of course happy to talk about this paper as well at the meeting.

Credible Autonomy Safety Argumentation
Philip Koopman, Aaron Kane, Jen Black
Carnegie Mellon University, Edge Case Research Pittsburgh, PA, USA 

Abstract   A significant challenge to deploying mission- and safety-critical autonomous systems is the difficulty of creating a credible assurance argument. This paper collects lessons learned from having observed both credible and faulty assurance argumentation attempts, with a primary emphasis on autonomous ground vehicle safety cases. Various common argumentation approaches are described, including conformance to a non-autonomy safety standard, proven in use, field testing, simulation, and formal verification. Of particular note are argumentation faults and anti-patterns that have shown up in numerous safety cases that we have encountered. These observations can help both designers and auditors detect common mistakes in safety argumentation for autonomous systems.

Download the full paper:
https://users.ece.cmu.edu/~koopman/pubs/Koopman19_SSS_CredibleSafetyArgumentation.pdf


How Many Operational Design Domains, Objects, and Events? Safe AI 2019 talk

Posted by badrut Friday, January 25, 2019 0 comments
Validating self-driving cars requires so, so much more than just "geo-fencing" if you want to make the problem tractable. My Safe AI 2019 paper and presentation explain and illustrate why this is the case.

Paper: https://users.ece.cmu.edu/~koopman/pubs/Koopman19_SAFE_AI_ODD_OEDR.pdf
Download slides: https://users.ece.cmu.edu/~koopman/pubs/koopman19_SAFE_AI_slides.pdf
(For slideshare version see below)

How Many Operational Design Domains, Objects, and Events
Phil Koopman & Frank Fratrik 

Abstract:
A first step toward validating an autonomous vehicle is deciding what aspects of the system need to be validated. This paper lists factors we have found to be relevant in the areas of operational design domain, object and event detection and response, vehicle maneuvers, and fault management. While any such list is unlikely to be complete, our contribution can form a starting point for a publicly available master list of considerations to ensure that autonomous vehicle validation efforts do not contain crucial gaps due to missing known issues.



How Road Testing Self-Driving Cars Gets More Dangerous as the Technology Improves

Posted by badrut Tuesday, January 15, 2019 0 comments
Safe road testing of autonomous vehicle technology assumes that human "safety drivers" will be able to prevent mishaps. But humans are notoriously bad at supervising autonomy. Ensuring that road testing is safe requires designing the test platform to have high "supervisability." In other words, it must be easy for a human to stay in the loop and compensate for autonomy errors, even when the autonomy is gets pretty good and the supervisor job gets pretty boring. This excerpt from a draft paper explains the concept and why it matters.

(update: full paper here:  https://users.ece.cmu.edu/~koopman/pubs/koopman19_TestingSafetyCase_SAEWCX.pdf
)
Figure 1.

An essential observation regarding self-driving car road testing is that it relies upon imperfect human responses to provide safety. There is some non-zero probability that the supervisor (a "safety driver") will not react in a timely fashion, and some additional probability that the supervisor will react incorrectly. Either of these outcomes could be an incident or mishap. Such a non-zero probability of unsuccessful failure mitigation means it is necessarily the case that the frequency of autonomy failures will influence on-road safety outcomes.

However, lower autonomy failure rates are not necessarily better. The types and frequencies of autonomy failures will affect the supervisability of the system. Therefore, the field failure rate and types of failures must be compatible with the measures being taken to ensure supervisor engagement. Thus, the failure profile must be “appropriate” rather than low.

Non-Linear Autonomy/Human Interactions

A significant difficulty in reasoning about the effect of autonomy failure on safety is that there is a non-linear response of human attentiveness to autonomy failure. We propose that there are five different regions of supervisability of autonomy failures, with two different hypothetical scenarios based on comparatively lower and higher supervisability trends illustrated in the figures.

1. Autonomy fails frequently in a dangerous way. In essence this is autonomy which is not really working. A supervisor faced with an AV test platform that is trying to run off the road every few seconds should terminate the testing and demand more development. We assume that such a system would never be operated on public roads in the first place, making a public risk assessment unnecessary. (Debugging of highly immature autonomy on public roads seems like a bad idea, and presents a high risk of mishaps.)

2. Autonomy fails moderately frequently but works or is benign most of the time. In this case the supervisor is more likely to remain attentive since an autonomy failure in the next few seconds or minutes is likely. The risk in this scenario is probably dominated by the ability of the supervisor to plan and execute adequate fault responses, and eventual supervisor fatigue.

3. Autonomy fails infrequently. In this case there is a real risk that the supervisor will lose focus during testing, and fail to respond in time or respond incorrectly due to loss of situational awareness. This is perhaps the most difficult situation for on-road testing, because the autonomy could be failing frequently enough to present an unacceptably high risk, but so infrequently that the supervisor is relatively ineffective at mitigation. This dangerous situation corresponds to the “valley of degraded supervision” in Figure 1.

4. Autonomy fails very infrequently, with high diagnostic coverage. At a high level of maturity, the autonomy might fail so infrequently that it is almost safe enough, and even a relatively disengaged driver can deal with failures well enough to result in a system that is overall acceptably safe. High coverage failure detection that prompts the driver to take over in the event of a failure might help improve the effectiveness of such a system. The ultimate safety of such a system will likely depend upon its ability to detect a risky situation with sufficient advance warning for the supervisor to re-engage and take over safely. (This scenario is generally aligned with envisioned production deployment of SAE Level 3 autonomy.)

5. Autonomy essentially never fails. In this case the role of the supervisor is to be there in case the expectation of “never fails” turns out to be incorrect in testing. It is difficult to know how to evaluate the potential effectiveness of a supervisor, other than that the supervisor will have the same tasks as the “very infrequently” preceding case, but is expected not to have to perform them.
Perhaps counter-intuitively, the probability of a supervisor failure is likely to increase as the autonomy failure rate decreases from regions 1 to 5 above (from left to right along the horizontal axis of the figures). In other words, the less often autonomy fails, the less reliable supervisor intervention becomes. The most dangerous operational region will be #3, in which the autonomy is failing often enough to present a significantly elevated risk, but not often enough to keep the supervisor alert and engaged. This is a well understood risk that must be addressed in a road testing safety case.

Figure 2 illustrates this effect with hypothetical performance data that results in an overall test
platform safety value in accordance with [math in the full paper]. A hypothetical lower supervisability curve results in a region in which the vehicle is less safe than a conventional vehicle driven by a human driver.

Safe testing requires a comparatively higher supervisability curve to ensure that the overall test platform safety is sufficiently high, as shown by Figure 2.

Figure 2.


Because autonomy capabilities are generally expected to mature over time, the safety argument must
be revisited periodically during test and development campaigns as the autonomy failure rate decreases from region 2 to 3 above. An intuitive – but dangerously incorrect – approach would be to assume that the requirements for test supervision can be relaxed as autonomy becomes more mature. Rather, it seems likely that the rigor of ensuring supervisors are vigilant and continually trained to maintain their ability to react effectively needs to be increased as autonomy technology transitions from immature to moderately mature. This effect only diminishes when the AV technology starts approximating the road safety of a conventional human driver all on its own (regions 4 & 5).

If you are actively doing self-driving car testing on public roads, please contact me for a preprint of the full paper that includes a GSN safety argumentation structure for ensuring road testing safety. I plan to present the full paper at SAE WCX 2019 in April.

-- Phil Koopman, Edge Case Research & Carnegie Mellon University

Three Safety Bins: road testing doesn't get you all the way to "safe"

Posted by badrut Tuesday, December 4, 2018 0 comments
There are three bins for self-driving car safety: 
(1) Obviously dangerous
(2) NOT obviously dangerous, and
(3) Safe.

Road testing and driving scenario coverage can help you get from bin 1 to bin 2.

Getting to Safe (bin 3) requires a whole lot more. That's because it requires handling rare, unexpected, and novel events in both the equipment and the environment.



Copyright © 2012 BEAUTIFUL VIEW. All rights reserved.

Super SEO created by Blogger Tuts | Published by GalleryBloggerTemplates.com