HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem

Yun Hua; Xiangfeng Wang; Bo Jin; Wenhao Li; Junchi Yan; Xiaofeng He; Hongyuan Zha

doi:10.1145/3447548.3467242

HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem

Yun Hua
, Xiangfeng Wang^*
, Bo Jin^*
, Wenhao Li
, Junchi Yan
, Xiaofeng He
, Hongyuan Zha

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

In spite of the success of existing meta reinforcement learning methods, they still have difficulty in learning a meta policy effectively for RL problems with sparse reward. In this respect, we develop a novel meta reinforcement learning framework called Hyper-Meta RL(HMRL), for sparse reward RL problems. It is consisted with three modules including the cross-environment meta state embedding module which constructs a common meta state space to adapt to different environments; the meta state based environment-specific meta reward shaping which effectively extends the original sparse reward trajectory by cross-environmental knowledge complementarity and as a consequence the meta policy achieves better generalization and efficiency with the shaped meta reward. Experiments with sparse-reward environments show the superiority of HMRL on both transferability and policy learning efficiency.

Original language	English
Title of host publication	KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher	Association for Computing Machinery
Pages	637-645
Number of pages	9
ISBN (Electronic)	9781450383325
DOIs	https://doi.org/10.1145/3447548.3467242
State	Published - 14 Aug 2021
Event	27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021 - Virtual, Online, Singapore Duration: 14 Aug 2021 → 18 Aug 2021

Publication series

Name	Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Conference

Conference	27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021
Country/Territory	Singapore
City	Virtual, Online
Period	14/08/21 → 18/08/21

Keywords

meta learning
reinforcement learning
sparse reward

Access to Document

10.1145/3447548.3467242

Cite this

Hua, Y., Wang, X., Jin, B., Li, W., Yan, J., He, X., & Zha, H. (2021). HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem. In KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 637-645). (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). Association for Computing Machinery. https://doi.org/10.1145/3447548.3467242

Hua, Yun ; Wang, Xiangfeng ; Jin, Bo et al. / HMRL : Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem. KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2021. pp. 637-645 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).

@inproceedings{49fd73e6f61c47c1ab4a894d8691197d,

title = "HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem",

abstract = "In spite of the success of existing meta reinforcement learning methods, they still have difficulty in learning a meta policy effectively for RL problems with sparse reward. In this respect, we develop a novel meta reinforcement learning framework called Hyper-Meta RL(HMRL), for sparse reward RL problems. It is consisted with three modules including the cross-environment meta state embedding module which constructs a common meta state space to adapt to different environments; the meta state based environment-specific meta reward shaping which effectively extends the original sparse reward trajectory by cross-environmental knowledge complementarity and as a consequence the meta policy achieves better generalization and efficiency with the shaped meta reward. Experiments with sparse-reward environments show the superiority of HMRL on both transferability and policy learning efficiency.",

keywords = "meta learning, reinforcement learning, sparse reward",

author = "Yun Hua and Xiangfeng Wang and Bo Jin and Wenhao Li and Junchi Yan and Xiaofeng He and Hongyuan Zha",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.; 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021 ; Conference date: 14-08-2021 Through 18-08-2021",

year = "2021",

month = aug,

day = "14",

doi = "10.1145/3447548.3467242",

language = "英语",

series = "Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",

publisher = "Association for Computing Machinery",

pages = "637--645",

booktitle = "KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining",

address = "美国",

}

Hua, Y, Wang, X, Jin, B, Li, W, Yan, J, He, X & Zha, H 2021, HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem. in KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, pp. 637-645, 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021, Virtual, Online, Singapore, 14/08/21. https://doi.org/10.1145/3447548.3467242

HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem. / Hua, Yun; Wang, Xiangfeng; Jin, Bo et al.
KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2021. p. 637-645 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - HMRL

T2 - 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021

AU - Hua, Yun

AU - Wang, Xiangfeng

AU - Jin, Bo

AU - Li, Wenhao

AU - Yan, Junchi

AU - He, Xiaofeng

AU - Zha, Hongyuan

PY - 2021/8/14

Y1 - 2021/8/14

N2 - In spite of the success of existing meta reinforcement learning methods, they still have difficulty in learning a meta policy effectively for RL problems with sparse reward. In this respect, we develop a novel meta reinforcement learning framework called Hyper-Meta RL(HMRL), for sparse reward RL problems. It is consisted with three modules including the cross-environment meta state embedding module which constructs a common meta state space to adapt to different environments; the meta state based environment-specific meta reward shaping which effectively extends the original sparse reward trajectory by cross-environmental knowledge complementarity and as a consequence the meta policy achieves better generalization and efficiency with the shaped meta reward. Experiments with sparse-reward environments show the superiority of HMRL on both transferability and policy learning efficiency.

AB - In spite of the success of existing meta reinforcement learning methods, they still have difficulty in learning a meta policy effectively for RL problems with sparse reward. In this respect, we develop a novel meta reinforcement learning framework called Hyper-Meta RL(HMRL), for sparse reward RL problems. It is consisted with three modules including the cross-environment meta state embedding module which constructs a common meta state space to adapt to different environments; the meta state based environment-specific meta reward shaping which effectively extends the original sparse reward trajectory by cross-environmental knowledge complementarity and as a consequence the meta policy achieves better generalization and efficiency with the shaped meta reward. Experiments with sparse-reward environments show the superiority of HMRL on both transferability and policy learning efficiency.

KW - meta learning

KW - reinforcement learning

KW - sparse reward

UR - https://www.scopus.com/pages/publications/85114935414

U2 - 10.1145/3447548.3467242

DO - 10.1145/3447548.3467242

M3 - 会议稿件

AN - SCOPUS:85114935414

T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

SP - 637

EP - 645

BT - KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

PB - Association for Computing Machinery

Y2 - 14 August 2021 through 18 August 2021

ER -

Hua Y, Wang X, Jin B, Li W, Yan J, He X et al. HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem. In KDD 2021 - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery. 2021. p. 637-645. (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). doi: 10.1145/3447548.3467242

HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this