AI chatbots like ChatGPT can simulate empathy and provide conversational support, but they still lack the ethical safeguards, contextual understanding, and accountability required for real therapy.
As the technology evolves, careful regulation and rigorous testing will be essential to ensure that AI helps people without causing unintended harm.
Experts warn that ChatGPT may pose ethical risks in therapy, including bias, misleading empathy, and unsafe mental health advice.
 |
| A person talking to an AI chatbot on a laptop for therapy-like conversation, digital brain and chat bubbles, warning symbols about ethics and safety. |
ChatGPT as a Therapist: New Study Reveals Serious Ethical Risks
Artificial intelligence tools like ChatGPT and other AI chatbots are increasingly being used for emotional support, self-reflection, and even therapy-style conversations.
Millions of people now turn to AI when they feel stressed, anxious, or overwhelmed. These tools are easy to access, available 24/7, and often feel non-judgmental.
But a new academic study suggests that relying on AI for mental health advice could come with serious ethical concerns.
Researchers from Brown University recently analyzed how AI chatbots behave when prompted to act like trained therapists. Their findings were troubling.
The systems were instructed to follow established psychotherapy techniques such as Cognitive Behavioral Therapy or Dialectical Behavior Therapy. The models repeatedly failed to meet professional mental health ethics standards set by organizations like the American Psychological Association.
The study identified 15 different ethical risks associated with AI counseling systems. These risks include mishandling crisis situations, reinforcing harmful beliefs, displaying bias, and using what researchers call “deceptive empathy.”
While AI could potentially expand access to mental health support, the research highlights why caution and proper regulation are urgently needed before AI therapy tools become widespread.
The Growing Trend of AI for Mental Health Support
Over the past few years, AI chatbots have evolved from simple question-answer tools to conversational assistants capable of discussing emotions, stress, and personal challenges.
Platforms like ChatGPT, Claude, and models based on Llama are increasingly used for mental health discussions.
Several factors explain why people are turning to AI for therapy-like conversations:
- Accessibility: AI tools are available anytime without appointments.
- Affordability: Therapy sessions can be expensive, while AI chatbots are often free or low-cost.
- Anonymity: Users may feel more comfortable sharing personal feelings with a chatbot than a human therapist.
Social media platforms such as TikTok, Reddit, and Instagram have also popularized “therapy prompts” that encourage people to ask AI for psychological guidance.
However, experts warn that AI may appear supportive without truly understanding human emotions or clinical context, which can lead to misleading advice.
The Brown University Study That Raised Concerns
The new research conducted by Brown University aimed to examine whether carefully designed prompts could guide AI chatbots to behave like ethical mental health professionals.
The findings were presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society, a major event focused on the ethical and social implications of artificial intelligence.
The research was led by computer science PhD candidate Zainab Iftikhar along with experts affiliated with Brown’s Center for Technological Responsibility, Reimagination and Redesign.
The study asked an important question:
Can AI chatbots safely provide therapy-like advice if they are prompted to behave like trained therapists?
The answer, according to the researchers, was largely no.
Even when given detailed instructions to follow established psychotherapy methods, the AI systems frequently violated professional mental health ethics standards.
How Prompts Try to Turn AI into Therapists
Many users attempt to transform AI chatbots into therapists using carefully written prompts. These prompts instruct the AI to behave like a specific type of mental health professional.
For example, users might type prompts such as:
- “Act as a cognitive behavioral therapist and help me reframe my negative thoughts.”
- “Use dialectical behavior therapy techniques to help me manage anxiety.”
- “Guide me through emotional regulation exercises.”
These prompts rely on well-known therapy methods like:
- Cognitive Behavioral Therapy (CBT), which focuses on changing negative thinking patterns.
- Dialectical Behavior Therapy (DBT), which helps people manage intense emotions and relationships.
While AI can generate responses that resemble these techniques, it does not actually understand psychological processes or patient history. Instead, it predicts responses based on patterns learned from large datasets.
This means the chatbot may sound helpful while lacking the deeper judgment and clinical expertise required for therapy.
How Researchers Tested AI Chatbots
To evaluate how AI behaves in therapy-like situations, the researchers designed a unique experiment.
Seven trained peer counselors with experience in Cognitive Behavioral Therapy conducted simulated counseling sessions with AI chatbots.
The models tested included:
- Versions of ChatGPT
- Claude
- Llama
These AI systems were prompted to act as CBT therapists during the sessions.
After the conversations were completed, the transcripts were reviewed by three licensed clinical psychologists. Their role was to identify potential ethical violations or unsafe responses.
The analysis revealed repeated patterns of problematic behavior, ultimately identifying 15 ethical risks in AI counseling systems.
The 15 Ethical Risks Identified in AI Therapy
AI therapy chatbots like ChatGPT are raising significant ethical concerns as they become more widely used for mental health support. Recent research led by Brown University highlights 15 distinct ethical risks when AI systems attempt to act like therapists. These include:
1. Lack of Contextual Understanding
AI chatbots often fail to consider a user’s personal background, culture, past experiences, and emotional history. Instead of tailoring responses, they provide generic advice that may not apply to the individual’s situation. This lack of contextual awareness can lead to misleading guidance and ineffective emotional support in complex mental health situations.
2. Overly Generic Advice
AI therapy responses frequently rely on generalized suggestions such as “practice mindfulness” or “stay positive.” While these suggestions sound helpful, they may oversimplify complex emotional struggles. Without detailed understanding of a user’s mental health condition, generic advice can feel dismissive and may fail to address deeper psychological concerns.
3. Reinforcing Harmful Beliefs
In some conversations, AI systems may unintentionally validate harmful beliefs expressed by users. For example, if someone shares negative assumptions about themselves or others, the chatbot might agree or fail to challenge those beliefs. This can reinforce unhealthy thought patterns rather than helping users develop healthier perspectives.
4. Poor Therapeutic Collaboration
Effective therapy requires collaboration between therapist and client. AI chatbots sometimes dominate the conversation by steering discussions toward predetermined solutions rather than encouraging exploration of emotions. This can limit meaningful dialogue and reduce the sense of partnership that is essential in professional psychotherapy sessions.
5. Deceptive Empathy
AI chatbots frequently use phrases like “I understand how you feel” or “I’m here for you.” While these statements sound supportive, the AI does not actually experience empathy or understand emotions. This creates an illusion of emotional connection that may mislead users into believing the chatbot genuinely cares about them.
6. Emotional Over-Attachment by Users
Because AI chatbots communicate in a friendly and supportive tone, users may develop emotional attachment to them. Some people might begin relying on the chatbot for emotional validation or decision-making. This dependency could reduce motivation to seek help from qualified mental health professionals.
7. Bias and Cultural Insensitivity
AI models are trained on large datasets that may contain biases related to gender, race, religion, or cultural norms. As a result, AI chatbots can sometimes produce responses that reflect these biases. In mental health discussions, culturally insensitive advice may alienate users or reinforce harmful stereotypes.
8. Inadequate Crisis Response
One of the most serious ethical risks occurs when users express suicidal thoughts or severe emotional distress. AI chatbots may respond with vague reassurance instead of directing users to emergency resources or professional support. Failure to handle crisis situations appropriately could lead to dangerous outcomes.
9. Avoidance of Sensitive Topics
Some AI systems attempt to avoid discussing highly sensitive mental health issues to reduce liability. When users bring up trauma, abuse, or suicidal ideation, the chatbot may refuse to engage or provide minimal responses. This avoidance can leave users feeling unsupported during critical moments.
10. Misinterpretation of User Intent
AI chatbots sometimes misunderstand user messages, especially when emotions are expressed indirectly or through sarcasm, slang, or cultural expressions. Misinterpreting a user’s emotional state may lead to inappropriate responses, which can worsen confusion or emotional distress rather than providing helpful guidance.
11. Lack of Professional Accountability
Human therapists operate under strict ethical guidelines and professional licensing rules from organizations like the American Psychological Association. AI systems, however, lack comparable oversight. When harmful advice is given, it is unclear who is responsible, creating a major accountability gap in AI-driven mental health tools.
12. False Sense of Professional Authority
AI chatbots may present responses in a confident tone that resembles expert advice. Users might assume the information comes from qualified mental health professionals, even though the AI is simply generating text patterns. This perceived authority can make users trust guidance that may not be clinically accurate.
13. Limited Understanding of Complex Disorders
Serious mental health conditions such as severe depression, trauma disorders, or personality disorders require specialized professional care. AI chatbots lack diagnostic capability and deep psychological understanding. Attempting to address these complex conditions through AI conversations alone could delay appropriate treatment from trained clinicians.
14. Privacy and Data Concerns
When users discuss sensitive emotional issues with AI chatbots, they may unknowingly share deeply personal information. If data storage or security practices are unclear, this information could be misused or exposed. Protecting user privacy is a major ethical challenge for AI systems handling mental health conversations.
15. Over-Reliance on AI Instead of Real Therapy
If users begin relying heavily on AI chatbots like ChatGPT for emotional support, they might delay or avoid seeking professional therapy. While AI can offer general guidance, it cannot replace trained therapists. Over-reliance could prevent people from receiving the specialized help they truly need.
The Accountability Gap in AI Counseling
Another major issue highlighted by the study is lack of accountability.
Human therapists operate under strict professional regulations. If a therapist behaves unethically, they can face consequences from licensing boards or legal authorities.
Organizations like the American Psychological Association set clear standards for mental health practice.
AI chatbots, however, operate in a largely unregulated environment.
If an AI system gives harmful mental health advice:
- Who is responsible?
- The developer?
- The company?
- The AI model itself?
Currently, there are no widely accepted legal or ethical frameworks governing AI therapy systems.
This regulatory gap makes it difficult to ensure safety and accountability.
Can AI Still Help With Mental Health?
Despite these concerns, the researchers emphasize that AI should not be completely dismissed in mental health care.
AI tools may still offer benefits such as:
- Providing basic emotional support
- Helping people track moods and habits
- Offering mental health education
- Reducing barriers to accessing information
In regions where therapy services are scarce or expensive, AI systems could help bridge some of the gaps.
However, experts agree that AI should function as a supportive tool rather than a replacement for licensed therapists.
Why Careful Evaluation of AI Systems Matters
Experts say the rapid development of AI technology has outpaced the systems used to evaluate its safety.
According to Ellie Pavlick, an AI researcher at Brown University, building AI systems is often much easier than thoroughly testing them.
Most AI models are evaluated using automated benchmarks rather than real-world human oversight.
The Brown study took more than a year and required collaboration between:
- AI researchers
- Mental health professionals
- Clinical psychologists
This kind of interdisciplinary evaluation is essential for understanding how AI behaves in sensitive areas like healthcare.
The Future of AI in Mental Health Care
Artificial intelligence has the potential to become a valuable tool in addressing the global mental health crisis. But researchers emphasize that safety must come first.
Future improvements may include:
- Stronger ethical guidelines for AI counseling systems
- Clear legal accountability frameworks
- Better bias detection and mitigation
- Improved crisis response mechanisms
- Human oversight in AI-assisted therapy tools
Ultimately, AI should be designed to support mental health professionals rather than replace them.
Until stronger safeguards are in place, experts recommend treating AI chatbots as informational tools—not as substitutes for professional psychological care.