Enhancing Portfolio Optimization via Heuristic-Guided Inverse Reinforcement Learning with Multi-Objective Reward and Graph-based Policy Learning

Portfolio optimization faces persistent challenges in adapting to dynamic market environments due to its dependence on static assumptions and high-dimensional decision spaces. Although reinforcement learning (RL) has emerged as a promising solution, conventional reward engineering methods often struggle to capture the complexities of market dynamics. Recent advances in deep RL and graph neural networks (GNNs) have sought to enhance market microstructure modeling. However, these methods still struggle with the systematic integration of financial knowledge. To address the above issues, we propose a novel heuristic-guided inverse RL framework for portfolio optimization. Specifically, our framework provides an effective mechanism for generating expert strategies that takes into account sector diversification and correlation constraints. Then, it employs a multi-objective reward optimization method to strike an adaptive balance between returns and risks. Furthermore, it utilizes heterogeneous graph policy learning with hierarchical attention mechanisms to model inter-stock relationships explicitly. Finally, we conduct extensive experiments on real-world financial market data to demonstrate that our framework outperforms several state-of-the-art baselines in terms of risk-adjusted returns. We also provide case studies to demonstrate the effectiveness of our framework in balancing return maximization and risk containment. Our code and data are publicly available at https://github.com/ChloeWenyiZhang/SmartFolio/.