Problem

The paper highlights that cognitive distortions significantly contribute to mental health disorders, and while Cognitive Behavioral Therapy (CBT) is effective in addressing these, its widespread application is hindered by a scarcity of trained therapists and high costs. Existing large language models (LLMs) explored for mental health support often lack domain-specific therapeutic grounding, tend to provide generic or overly agreeable responses, and fail to incorporate well-defined annotations for cognitive distortions, preventing effective cognitive restructuring. Current evaluation methods for mental health dialogue systems also fall short, typically relying on general NLP metrics or basic user satisfaction, rather than clinically relevant criteria for CBT.

Method

Cognivia is an evidence-based AI therapist developed through a multi-stage process. First, an expert seed set of distorted thoughts, cognitive distortion categories, and rational responses is curated from authoritative CBT literature. This seed data provides structured examples of cognitive restructuring patterns. Second, this dataset is augmented using multi-stage prompting and structured generation strategies on a large-scale psychological question-answering dataset (PsyQA). Specifically, DeepSeek identifies cognitive distortions in PsyQA questions, and then ChatGPT (GPT-5 Mini) generates CBT-aligned rational responses. Third, a lightweight LLM (Qwen2.5-7B-Instruct) is fine-tuned using Low-Rank Adaptation (LoRA) on this augmented CBT Cognitive Triplet Dataset to create Cognivia. The paper also proposes CogEval, a hierarchical quality evaluation framework with four dimensions—semantic fidelity, robustness and fault tolerance, deployment feasibility with user adoption, and relational boundary integrity—to rigorously assess the effectiveness of generated rational responses, developed in collaboration with behavioral science experts.

Results

Extensive evaluations using automatic NLP metrics (BLEU, ROUGE), LLM-based judges with both existing and the proposed CogEval criteria, and human assessments by ten behavioral science experts consistently demonstrate Cognivia's superior performance. Fine-tuning with the proposed dataset significantly improved lexical metrics, with Cognivia achieving performance comparable to GPT-5 Mini and outperforming strong open-source baselines like LLaMA-3.3, Mistral Medium 3, and Qwen3. Under LLM-as-a-judge evaluations, Cognivia achieved top scores in relevance, CBT structure, and helpfulness, and scored highest on 9 out of 12 CogEval metrics. Human expert evaluations further confirmed Cognivia's superior overall response quality, with higher median and mean scores than baseline models. An ablation study also showed the critical role of the expert seed set in Cognivia's performance. The CogEval framework itself was validated by experts, with high ratings for reasonableness and coverage completeness.

Implications

Cognivia represents a promising step towards scalable and accessible CBT-oriented mental health assistance by integrating evidence-based psychological knowledge with advanced LLMs. The framework operationalizes CBT through accurate cognitive distortion identification and the generation of therapeutically aligned rational responses, offering supportive and empathetic guidance. While effective, the current framework acknowledges limitations such as not explicitly accounting for individual differences (cultural background, symptom severity) and being designed as supportive guidance rather than a substitute for professional care. Future work includes extending the system to multi-turn dialogue, presenting long responses in more user-friendly formats, and conducting real-world intervention studies in collaboration with mental health professionals to enhance external validity and address ethical safeguards.