How we think about safety for users experiencing mental or emotional distress
We recognize that people may turn to our products during moments of mental or emotional distress, and we take seriously the responsibility of providing helpful, safe, and respectful responses in those moments. Our approach is grounded in a commitment to support users without causing harm, while also being transparent about the current limitations of our systems.
Our guiding principles
We design our systems with the following principles in mind:
- Prioritize user safety and well-being — We aim to avoid generating content that could escalate distress, encourage self-harm, or provide harmful advice.
- Encourage professional help — When appropriate, we direct users to qualified mental health resources and crisis support services.
- Respect user autonomy — We strive to offer non-judgmental, empathetic, and balanced responses that acknowledge the user’s experience.
- Be transparent about limitations — We clearly communicate that our systems are not a substitute for professional diagnosis, treatment, or crisis intervention.
Limits of today’s systems
Despite our best efforts, current systems have significant limitations. They may misinterpret nuanced emotional cues, fail to recognize the severity of a crisis, or provide responses that feel generic or unhelpful. In some cases, they may inadvertently reinforce negative thought patterns or offer overly clinical language that lacks warmth.
We do not claim that our systems can understand or replicate human empathy. They are tools, not clinicians, and they cannot fully grasp the complexity of a person’s lived experience.
These limitations mean that our systems are not designed to handle all mental health scenarios, and we advise users to seek human support when they need it.
Work underway to refine our systems
We are actively investing in research and engineering to improve how our systems handle mental health-related conversations. This includes:
- Collaborating with clinical experts — We work with psychiatrists, psychologists, and crisis intervention specialists to inform our safety guidelines and response strategies.
- Improving detection of distress signals — We are developing better models to recognize when a user may be in acute distress or at risk of self-harm.
- Refining response quality — We are iterating on language generation to make responses more compassionate, context-aware, and actionable.
- Building safer fallback mechanisms — When uncertainty is high, we are testing ways to default to safer, more conservative responses, including direct referrals to crisis lines.
- User feedback and evaluation — We continuously collect feedback from users and conduct red-team exercises to identify failure modes and improve our systems.
We are committed to making steady, measurable progress in this area, and we will continue to share updates as our work evolves.