Paper List

Journal: ArXiv Preprint
Published: Unknown
Human-Computer InteractionMental Health Informatics

Human-Centred Evaluation of Text-to-Image Generation Models for Self-expression of Mental Distress: A Dataset Based on GPT-4o

School of Culture and Communication, Swansea University, United Kingdom | Department of Informatics, University of Oslo, Norway

Sui He, Shenbin Qian
Figure
Figure
Figure
Figure
Figure

The 30-Second View

IN SHORT: This paper addresses the critical gap in evaluating how AI-generated images can effectively support cross-cultural mental distress communication, particularly for international students facing linguistic and cultural barriers.

Innovation (TL;DR)

  • Methodology Introduces the first publicly available text-to-image evaluation dataset with human judgment scores specifically for mental health communication, comprising 100 textual descriptions, 400 AI-generated images, and 400 categorical evaluation scores.
  • Methodology Develops and evaluates four persona-based prompt templates (basic, illustrator, photographer, creative artist) rooted in contemporary counselling practices, with the illustrator persona achieving the highest total helpfulness score (284 out of possible 600).
  • Biology Demonstrates that AI-generated images can facilitate self-expression of mental distress, with 44% of images rated as 'slightly helpful' and 27% as 'helpful', achieving a mean helpfulness score of 2.4 on a 0-6 scale.

Key conclusions

  • The illustrator persona prompt achieved the highest total helpfulness score (284) and was selected as the 'best' image in 31% of cases, significantly outperforming other prompts (basic: 252, creative artist: 218, photographer: 210).
  • Human evaluation shows minimal correlation with automatic semantic alignment metrics (Spearman's ρ=0.0271, Kendall's τ=0.0201), highlighting the need for emotion-aware evaluation frameworks beyond traditional similarity measures.
  • AI-generated images demonstrated positive utility for mental distress expression, with 71% of images rated as at least 'slightly helpful' (score ≥2) and only 29% rated as 'not helpful' (score=0).
Background and Gap： Despite growing interest in generative AI for emotional communication, there is a lack of systematically constructed datasets combining authentic user-generated descriptions of mental distress, AI-generated images, and human evaluation scores reflecting perceived helpfulness for self-expression.

Abstract: Effective communication is central to achieving positive healthcare outcomes in mental health contexts, yet international students often face linguistic and cultural barriers that hinder their communication of mental distress. In this study, we evaluate the effectiveness of AI-generated images in supporting self-expression of mental distress. To achieve this, twenty Chinese international students studying at UK universities were invited to describe their personal experiences of mental distress. These descriptions were elaborated using GPT-4o with four persona-based prompt templates rooted in contemporary counselling practice to generate corresponding images. Participants then evaluated the helpfulness of generated images in facilitating the expression of their feelings based on their original descriptions. The resulting dataset comprises 100 textual descriptions of mental distress, 400 generated images, and corresponding human evaluation scores. Findings indicate that prompt design substantially affects perceived helpfulness, with the illustrator persona achieving the highest ratings. This work introduces the first publicly available text-to-image evaluation dataset with human judgment scores in the mental health domain, offering valuable resources for image evaluation, reinforcement learning with human feedback, and multi-modal research on mental health communication.

Code