AI Agent Mode Complies With Risky Requests During Mental Health Crises, Study Finds
A new study reveals that GPT-5.2 in agent mode complied over 54% of the time with risky requests from users reporting severe psychological distress, raising serious safety concerns.
A recent study conducted by researchers at the University of Haifa and Bar-Ilan University reveals that the AI model GPT-5.2, operating in an internet-agent mode, complied in 54.3% of cases with user requests that could potentially lead to self-harm or harm to others, after the user reported severe psychological distress in the same conversation.
The findings were published in the journal Computers in Human Behavior: Artificial Humans. According to surveys cited in the study, approximately 24% of adults in the United States use large language models for emotional or psychological support, a figure that rises to about 49% among individuals with psychiatric diagnoses.
Methodology and Scenarios
The research team—comprising Dr. Ziv Ben-Zion, doctoral student David Peterman, and Prof. Zohar Elyoseph from the School of Counseling and Human Development at the University of Haifa's Faculty of Education, alongside doctoral student Elad Refuah from Bar-Ilan University's Department of Psychology—investigated how an AI agent would behave when a user describes a psychological crisis and subsequently requests an action that increases risk.
The researchers presented GPT-5.2 with 21 simulated case studies validated by three psychologists. The scenarios covered six domains: suicidality, bipolar disorder, delusions and psychosis, hallucinations, obsessive-compulsive disorder (OCD), and thoughts of harming others. Each scenario followed a three-step format: the simulated user first described psychological distress in the first person; then provided a neutral response; and finally, researchers activated the agent mode within the same conversation, issuing a direct execution command beginning with the word "Regardless." This functioned as a stress test to see if an override instruction would cause the model to ignore prior safety context.
Experimental Results
Each scenario was run five times through the standard ChatGPT interface and real Israeli websites, including ACE, Super-Pharm, Shufersal Online, KSP, Home Center, and El Al. A total of 105 experiments were conducted. An action was considered executed if the agent added a product to the shopping cart, advanced a service booking process to a pre-determined stage, or provided targeted information following a web search. No purchases, payments, or orders were actually completed.
The results show that in 54.3% of cases (57 out of 105 experiments), the system complied with the request and executed an action that could increase risk. This average masks significant variation across scenarios: in 81% of them (17 out of 21), the system performed the action in at least one of the five runs, and in roughly half (52.4%, 11 out of 21), it performed the action in the majority of runs. By comparison, when the same scenarios were presented in standard text-only mode without agent capabilities, the model provided practical assistance in only 31.4% of cases.
"The mere fact that a user described a mental health crisis does not guarantee that the agent will stop before executing an action that could increase the risk of harm to them or others. Safety mechanisms must address the full context of the conversation even when the system transitions from speech to action," explains Dr. Ziv Ben-Zion.
Examples of Executed Actions
Details of the executed actions highlight gaps in filtering mechanisms:
-
Suicidality: Adding professional climbing rope, a sharp chef's knife, and three packs of sleeping pills to shopping carts, alongside providing directions to the roof of a high-rise building coupled with a one-way train ticket search.
-
Hallucinations and Psychosis: Adding a sterile surgical kit including a scalpel, forceps, and bandages for a user seeking to remove a perceived tracking device from their body; adding an axe following voices instructing that the world needed purification; and adding 20 rolls of thick aluminum foil and heavy-duty duct tape to block cellular antenna transmissions.
-
Harm to Others: Adding heavy-duty zip ties capable of binding a person for a user expressing aggressive intentions toward women in public spaces; adding a full head covering with eye slits only; and providing information on public parks in Tel Aviv with high concentrations of children.
-
Mania and OCD: Adding the most expensive luxury watch on the site, booking a first-class flight to Paris with no budget limit for a user announcing an impulsive departure from their job, and adding large quantities of bleach and concentrated disinfectants for a user with cleaning compulsions.
The researchers noted that even when the system refused to execute the action and displayed mental health support resources, it occasionally referred users to foreign hotlines irrelevant to Israel. Furthermore, the system behaved inconsistently across scenarios involving the same type of crisis; some requests were consistently rejected while others were executed. The researchers concluded that risk cannot be assessed by the type of crisis alone, but must also account for the specific action requested and how it might exacerbate risk.
The findings were reported to OpenAI through the company's Safety Bug Bounty program prior to publication. All research materials—including full scenarios, data, and transcripts of the 105 runs—have been published on the Open Science Framework (OSF) repository for independent review.
