CFP: A Reinforcement Learning Framework for Comprehensive Fairness-Performance Trade-Off in Machine Learning

Machine learning models are increasingly used for impactful decisions, such as loan approval, criminal sentencing, and resume filtering, raising concerns about ensuring fairness without sacrificing performance. However, fairness has multiple definitions, and existing techniques targeting specific metrics have limitations in improving multiple notions of fairness simultaneously. In this work, we establish a comprehensive measurement to simultaneously consider multiple fairness notions as well as performance, and propose new metrics through an in-depth analysis of the relationship between different fairness metrics. Based on the comprehensive measurement and new metrics, we present CFP, a reinforcement learning-based framework, to efficiently improve the fairness-performance trade-off in machine learning classifiers. We conduct extensive experiments to evaluate CFP on 6 tasks, 3 machine learning models, and 15 fairness-performance measurements. The results demonstrate that CFP can improve the classifiers on multiple fairness metrics without sacrificing its performance.