Offline minimax Q-function learning for undiscounted indefinite-horizon MDPs

Fengying Li; Yuqiang Li; Xianyi Wu; Wei Bai

doi:10.1007/s10463-025-00924-1

Offline minimax Q-function learning for undiscounted indefinite-horizon MDPs

Fengying Li, Yuqiang Li, Xianyi Wu, Wei Bai

School of Statistics

Ningxia Normal University

Research output: Contribution to journal › Article › peer-review

Abstract

This work considers the offline evaluation problem for indefinite-horizon Markov Decision Processes. A minimax Q-function learning algorithm is proposed, which, instead of i.i.d. tuples (s,a,s′,r), evaluates undiscounted expected return based by i.i.d. trajectories truncated at a given time step. The confidence error bounds are developed. Experiments using Open AI’s Cart Pole environment are employed to demonstrate the algorithm.

Original language	English
Pages (from-to)	535-562
Number of pages	28
Journal	Annals of the Institute of Statistical Mathematics
Volume	77
Issue number	4
DOIs	https://doi.org/10.1007/s10463-025-00924-1
State	Published - Aug 2025

Keywords

Indefinite-horizon MDPs
Minimax Q-function learning
Occupancy measure
Off-policy
Policy evaluation

Access to Document

10.1007/s10463-025-00924-1

Cite this

@article{7bb8586193954a7d987a05551d380e85,

title = "Offline minimax Q-function learning for undiscounted indefinite-horizon MDPs",

abstract = "This work considers the offline evaluation problem for indefinite-horizon Markov Decision Processes. A minimax Q-function learning algorithm is proposed, which, instead of i.i.d. tuples (s,a,s′,r), evaluates undiscounted expected return based by i.i.d. trajectories truncated at a given time step. The confidence error bounds are developed. Experiments using Open AI{\textquoteright}s Cart Pole environment are employed to demonstrate the algorithm.",

keywords = "Indefinite-horizon MDPs, Minimax Q-function learning, Occupancy measure, Off-policy, Policy evaluation",

author = "Fengying Li and Yuqiang Li and Xianyi Wu and Wei Bai",

note = "Publisher Copyright: {\textcopyright} The Institute of Statistical Mathematics, Tokyo 2025.",

year = "2025",

month = aug,

doi = "10.1007/s10463-025-00924-1",

language = "英语",

volume = "77",

pages = "535--562",

journal = "Annals of the Institute of Statistical Mathematics",

issn = "0020-3157",

publisher = "Springer Netherlands",

number = "4",

}

TY - JOUR

T1 - Offline minimax Q-function learning for undiscounted indefinite-horizon MDPs

AU - Li, Fengying

AU - Li, Yuqiang

AU - Wu, Xianyi

AU - Bai, Wei

N1 - Publisher Copyright: © The Institute of Statistical Mathematics, Tokyo 2025.

PY - 2025/8

Y1 - 2025/8

N2 - This work considers the offline evaluation problem for indefinite-horizon Markov Decision Processes. A minimax Q-function learning algorithm is proposed, which, instead of i.i.d. tuples (s,a,s′,r), evaluates undiscounted expected return based by i.i.d. trajectories truncated at a given time step. The confidence error bounds are developed. Experiments using Open AI’s Cart Pole environment are employed to demonstrate the algorithm.

AB - This work considers the offline evaluation problem for indefinite-horizon Markov Decision Processes. A minimax Q-function learning algorithm is proposed, which, instead of i.i.d. tuples (s,a,s′,r), evaluates undiscounted expected return based by i.i.d. trajectories truncated at a given time step. The confidence error bounds are developed. Experiments using Open AI’s Cart Pole environment are employed to demonstrate the algorithm.

KW - Indefinite-horizon MDPs

KW - Minimax Q-function learning

KW - Occupancy measure

KW - Off-policy

KW - Policy evaluation

UR - https://www.scopus.com/pages/publications/105003009213

U2 - 10.1007/s10463-025-00924-1

DO - 10.1007/s10463-025-00924-1

M3 - 文章

AN - SCOPUS:105003009213

SN - 0020-3157

VL - 77

SP - 535

EP - 562

JO - Annals of the Institute of Statistical Mathematics

JF - Annals of the Institute of Statistical Mathematics

IS - 4

ER -

Offline minimax Q-function learning for undiscounted indefinite-horizon MDPs

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this