Towards Systematically Engineering Autonomous Systems Using Reinforcement Learning and Planning

Abstract

Autonomous systems need to be able dynamically adapt to changing requirements and environmental conditions without redeployment and without interruption of the systems functionality. The EU project ASCENS has developed a comprehensive suite of foundational theories and methods for building autonomic systems. In this paper we specialise the EDLC process model of ASCENS to deal with planning and reinforcement learning techniques. We present the “AIDL” life cycle and illustrate it with two case studies: simulation-based online planning and the PSyCo reinforcement learning approach for synthesizing agent policies from hard and soft requirements. Related work and potential avenues for future research are discussed.

Dedicated to Manuel Hermenegildo.

A Markov Decision Processes

A Markov Decision Process (MDP) M defines a domain as a set S of states consisting of all states of the environment and the agent, a set of A of agent actions, and a probability distribution $T : p(S \vert S, A)$ describing the transition probabilities of reaching some successor state when executing an action in a given state. For expressing optimisation goals the labelled transition system is extended by a reward function $R : S \times A \times S \rightarrow \mathbb {R}$ which gives the expected immediate reward gained by the agent for taking each action in each state. Moreover, an initial state distribution $\rho : p(S)$ is given.

An episode $\textbf{e} \in E$ is a finite or infinite sequence of transitions $(s_i, a_i, s_{i + 1}, r_i)$, $s_i, s_{i + 1} \in S$, $a_i \in A, r_i = R(s_i, a, s_{i + 1})$ in the MDP. For a given discount parameter $\gamma \in [0,1]$ and any finite or infinite episode $\textbf{e}$, the cumulative return $\mathcal {R}$ is the discounted sum of rewards $\mathcal {R} = \sum _{i = 1}^{|\textbf{e}|} \gamma ^{i} r_i$. Depending on the application, the agent behaves in an environment according to a memoryless stationary policy $\pi : S \rightarrow p(A)$ or according to a deterministic memoryless policy $\pi : S \rightarrow A$ with the goal to maximise the expectation of the cumulative return $\mathbb {E}(\mathcal {R})$.

A partially observable Markov Decision Process (POMDP) [32] is a Markov decision process together with a set $\varOmega$ of observations and an observation probability distribution $O : p(\varOmega \vert S, A)$.

A Constrained Markov Decision Process (CMDP) has an additional cost function $C : S \times A \times S \rightarrow \mathbb {R}$ which can be used for expressing constraints and safety goals.