Patients around the world are increasingly seeking medical advice from artificial intelligence systems. In a recent Times of India report, after several specialists failed to identify the cause of a woman’s chronic cough and internal bleeding, her daughter turned to ChatGPT. The system asked a question that clinicians had not asked: whether the patient was taking a blood pressure medication known to cause such symptoms. When the medication was changed, his condition improved.
When empathy and fluency shape trust
Episodes like this are becoming more common as AI tools enter everyday healthcare, in part because they are seen as more attentive: responding immediately, asking follow-up questions without visible impatience, and expressing conclusions with ease and confidence.
A 2023 study published in JAMA Internal Medicine found that patients viewed AI-generated medical responses as significantly more empathetic and trustworthy than responses from doctors. For people facing rushed visits and fragmented care, interactions with AI may feel emotionally safer than clinical encounters.
But mastery is not the same as expertise. Under stress, patients may mistake a polite tone for accuracy. AI is fast enough to shorten thinking and smooth enough to discourage questioning. When it fills gaps in access, time, and empathy, it risks becoming not only a source of information, but also a substitute for clinical judgment.
Clinicians are not immune to a similar dynamic. The adoption of AI tools is accelerating in the healthcare industry. OpenEvidence, a medical AI company, estimates that more than 100 million Americans will be treated this year by doctors using AI-based tools. Ambient documentation systems, a commonly used tool, reduce administrative burden and allow doctors to focus on their patients. Yet early evidence suggests that AI scribes can flatten nuance, failing to capture distress or psychosocial context. In high-pressure environments, peer-reviewed studies show that radiologists may rely more than expected on AI triage labels, a form of automation bias that can contribute to delayed care.
These failures are rarely due to negligence. They appear when tools are introduced into workflows without clear boundaries on responsibilities. The lack of guardrails allows errors to propagate quickly.
Individual vigilance is not enough. As AI becomes increasingly integrated into care, the signals that shape trust and authority are evolving faster than our oversight structures can adapt. Good results in controlled environments are only part of the picture. The more difficult question is whether AI is integrated into clinical workflows in a way that keeps responsibility, authority, and accountability properly aligned.
Design for judgment, not just accuracy
Computational humility is an emerging framework for addressing these challenges. This approach designs systems that highlight uncertainty, make model limitations visible, and preserve human judgment rather than obscuring it. This means asking whether an AI is accurate and deciding when its results should be questioned, overturned, or ignored. This requires attention to technical performance, emotional vulnerabilities of patients, professional autonomy of clinicians, and the real-world context in which decisions are made.
It also means aligning financial incentives with patient-centered outcomes. Value-based payment models, such as Medicare’s shared savings and value-based purchasing programs, reward health systems for reducing readmissions and improving chronic disease outcomes rather than increasing the volume of services. In such environments, AI tools are evaluated not only on the basis of their technical performance, but also on their ability to improve real-world outcomes, workflow usability, and patient safety. When reimbursement and responsibility are shared among developers, providers, and payers, the incentive shifts from rapid deployment of tools to their responsible integration.
AI is already integrated into healthcare. What remains unresolved is the relationship that modern medicine is prepared to establish with it: a relationship based on fluidity and convenience, or a relationship structured around clarity, boundaries, and shared responsibility.
Photo: Irina_Strelnikova, Getty Images
Leslie Pascaud
Leslie Pascaud is a leader in strategic analytics and marketing with over 35 years of experience growing B2B, B2C, and nonprofit organizations in health technology and global health. She specializes in translating complex innovations into clear, differentiated narratives that resonate with business and mission-driven audiences. As CMO of a digital clinical trials company, she led cross-functional teams to develop thought leadership and go-to-market strategies that contributed to three consecutive years on the Inc. 5000 list of fastest-growing companies.
She is a strategic advisor at Kinetic Strategic Consulting Group and three-term board advisor to Tiko, 2025 recipient of the Audacious Project grant for its “big, bold solution” via a pioneering digital platform that provides access to health services for young African women. Leslie’s current work focuses on the evolving role of artificial intelligence in healthcare, particularly how AI systems influence clinical judgment, patient trust, and alignment of responsibilities in care delivery.
This message appears via the MedCity Influencers program. Anyone can post their views on healthcare business and innovation on MedCity News through MedCity Influencers. Click here to find out how.