Self-reflective experiential learning for persistent autonomy

Abstract

Robots have been expected to achieve persistent autonomy for a long time, in which robots are required to safely operate in unknown environments for extended lengths of time while without human interventions. Reinforcement learning holds the promise for persistent autonomy because it can adapt to dynamic and unstructured environments by automatically learning optimal policies from the interactions between robots and environments. However, failures can be unavoidable in the learning process as reinforcement learning can learn the outcome of an action only by executing the action itself. These failures can cause damages to robots or environments in practical applications and hence hinder persistent autonomy. In general, human interventions are usually needed to avoid or resolve the learning failures, but they can be unavailable in practical applications such as space exploration, search and rescue, and underground or underwater construction. Based on a multi-level architecture for persistent autonomy, this dissertation proposes new self-reflective, experiential strategies and methods, aiming at achieving safe adaptions to different environments with minimum human interventions. At the strategic level, while understanding learning is not always necessary and beneficial, this dissertation adds a high-level sophistication of whether and when to learn to reduce failures during learning. At the tactical level, this dissertation proposes a new self-recoverable reinforcement learning algorithm that consists of a multi-state recovery strategy and a failure-prevention strategy. The multi-state recovery strategy improves the learning’s own capability of handling already occurred failures and the failure-prevention strategy learns from failures that are usually ignored and abandoned before to generate a more effective strategy of preventing future similar failures. At the operational level, this dissertation proposes a new multi-objective-optimization-based auto-tuning method to adjust control parameters for robustly achieving the upper-level learning behaviors.