Autonomous systems are prone to errors and failures without knowing why. In critical domains like driving, these autonomous counterparts must be able to recount their actions for safety, accountability, and trust. An explanation: a model-dependent reason or justification for the decision of the autonomous agent being assessed, is a key component for post-mortem failure analysis, but also for pre-deployment verification. I will present neuro-symbolic systems that use neural networks and commonsense knowledge to detect and explain unreasonable vehicle scenarios, even if the autonomous vehicle has not seen that error before. In the second part of the talk, I will motivate the use of explanations as a testing framework for autonomous systems. I will conclude by discussing new challenges at the intersection of explainable AI and autonomy toward autonomous vehicles systems that are explainable by design.