PCDialogEval: Persona and Context Aware Emotional Dialogue Evaluation

Endowing dialogue systems with emotional intelligence is an essential strategy for machines to achieve deep social interaction with users, for which effective evaluation metrics for emotional dialogue are in urgent need. However, most existing evaluation methods ignore the impact of users’ individual differences and situational contexts on emotional expressions, which poses a significant challenge in assessing the emotional expression capabilities of dialogue models. To address these issues, we propose a novel evaluation model that incorporates personality into evaluation metrics for dialogue systems. Our model quantifies the influence of human personality on emotional expressions and simulates the emotional transfer during conversations to calculate the intensity of emotional expressions in candidate sentences. To accomplish this calculation, we first incorporates “Big Five Personality” traits for personality analysis, and subsequently modify the emotion vector in a Valence-Arousal-Dominance (VAD) space. Furthermore, we construct mood transfer equations to simulate the impact of the conversational context on emotional expressions. Additionally, we propose an additional assessment at both sentence and session-levels to evaluate the fluency and coherence of the generated dialogue. Experimental results on two datasets demonstrate the effectiveness of the proposed evaluation model in accurately assessing the emotional expression capabilities of dialogue systems.