Transfer reinforcement learning via meta-knowledge extraction using auto-pruned decision trees

摘要

Transfer reinforcement learning (RL) has recently received increasing attention to make RL agents have better learning performance in target Markov decision problems (MDPs) by using the knowledge learned in source MDPs. However, it is still an open and challenging problem to improve the transfer capability and interpretability of RL algorithms. In this paper, we propose a novel transfer reinforcement learning approach via meta-knowledge extraction using auto-pruned decision trees. In source MDPs, pre-trained policies are firstly learned via RL algorithms using general function approximators. Then, a meta-knowledge extraction algorithm is designed with an auto-pruned decision tree model, where the meta-knowledge is learned by re-training the auto-pruned decision tree based on the data samples generated from the pre-trained policies. The state spaces of meta-knowledge are determined by estimating the uncertainty of state–action pairs in pre-trained policies based on the entropy value of leaf nodes. In target MDPs, according to whether the state is in the state set of meta-knowledge, a hybrid policy is generated by integrating the meta-knowledge and the policies learned on the target MDPs. Based on the proposed transfer RL approach, two meta-knowledge-based transfer reinforcement learning (MKRL) algorithms are developed for MDPs with discrete action spaces and continuous action spaces, respectively. Experimental results in several benchmark tasks show that the MKRL algorithm outperforms other baselines in terms of learning efficiency and interpretability in the target MDPs with generic cases of task similarity.