How Hospitals’ Use of GenAI is Putting Patients at Risk Without Realizing It
The accelerating advancements in artificial intelligence are challenging traditional business assumptions across industries. This article will focus on large language models, also known as Generative AI (GenAI) platforms.
How Hospitals’ Use of GenAI is Putting Patients at Risk Without Realizing It
The accelerating advancements in artificial intelligence are challenging traditional business assumptions across industries. This article will focus on large language models, also known as Generative AI (GenAI) platforms. This can be clearly seen in the healthcare information technology space, specifically the provider electronic healthcare records (EHRs), where there is a broad opportunity for workflow efficiency, experience optimization, and personalization leveraging GenAI. But the question is – at what potential risk? Healthcare organizations must evaluate AI-based solutions and services to improve their care delivery, but they need to avoid the “move fast and break things” mentality due to patient safety risk.One key concern with public GenAI models is hallucinations, which can potentially return false positives or false negatives in responses to prompts. With false negatives, we may run into patient safety risks and could miss the early stages of a condition, resulting in worse outcomes and higher cost of treatment. With false positives, there remains the financial risk of overtreatment, potentially without reimbursement, but also the introduction of undue risk from excessive treatment or services. Hallucinations, among other concerns, should be mitigated before GenAI solutions are deployed in patient or provider facing environments.Based on the need to leverage Gen AI and risk from issues like hallucinations, this article will describe the four primary ways organizations engage with AI models, the vulnerabilities or potential threats associated with each, and suggest varying levels of governance for internal and vendor practices, based on the risks.The Four Basic AI Implementation Approaches for Hospitals & Health Systems Here is a framework of four primary implementation approaches for GenAI, four key parameters and example use-cases.1. Free Public ModelsLoading up a GenAI instance in a browser takes a clinician seconds, and with a few words, they are off to the races. The concern is the unbalanced risk and reward ratio: It is easy to upload a patient’s after-visit summary and have a consumer-grade generative AI model like ChatGPT summarize it in a “clinically accurate, helpful tone, for an audience that has completed high school.”There are several cyber/compliance issues with this approach:HIPAA and exposing PHI compliance issuesClinical accuracy of the modelInherent biases that are baked in and influence outputs based on patient age, sex, gender, and medical historyModel hallucinations that can make inferences based on inappropriate training dataEach issue disqualifies this approach from being acceptable in a clinical context. At this stage, for hospitals and health systems, there are no acceptable clinical use-cases for these models when it comes to patient care or back-office use-cases, as they cannot accommodate PHI or PII, and sharing sensitive data into a model that retains context across conversations may constitute a breach. There may be some acceptable, non-clinical use cases, but the need for governance is high.2. Private Instances Leveraging Public Model InfrastructureTo gain the full benefits of Gen AI, healthcare providers need options that can accommodate the organization’s clinical data. Licensed or private instances of public models can handle this constraint if configured correctly, as there are settings to disable submitted data from returning to the source model as training data. However, that setting must be enabled, and doing so mitigates some of the benefits of the GenAI, such as retaining context across conversations.The cyber/compliance issues with this approach are:The outputs of these models must still be validated by a clinician with a critical eye, as this approach does not address the same issues listed above.BiasesClinical accuracyHallucinationsWhere this approach can be helpful, however, is in using GenAI as collaboration tools and brainstorming partners. Again, clinical knowledge cannot be taken for granted, but having a model that can accommodate clinical context (albeit restricted to that conversational session) is a powerful tool that can speed up and improve clinical decision-making and care planning.For organizations looking to provide “something” for their team to use broadly, this approach is typically the best place to start. However, this is where the need for governance is highest. It is very easy to misuse these tools and trust their outputs inappropriately.3. Embedded SDK (Software Development Kit) / API (Application Programming Interface)As we leave the public models and move to more technical use-cases for application interaction, there is a broader set of applicable use-cases. Organizations leveraging embedded SDKs, APIs, or implementing AI models directly into precise workflows reduce decision-making for end users and constrain the use of Gen AI to scenarios deemed appropriate or safe. This is basically leveraging GenAI behind a user interface that has a specific function.This requires a significant step up in technical complexity as we are no longer discussing licensing ChatGPT and “configuring it” to not persist PHI. In this implementation, organizations are augmenting core systems of record and workflow, typically EHRs, to achieve operational efficiencies. Commonly, this would be done by leveraging an AI-model from a marketplace partner (optimally, your EHRs marketplace) or bolt-on point solution. Two of the most common use-cases in healthcare provider and EHR technical ecosystems today are scribe-related note transcription and medical coding assistance.Some potential cyber/compliance issues for this method are:Compliance issues based on systems that PHI crossesClinical Accuracy and the ability to audit resultsVendor dependency in a rapidly changing marketplace4. Bespoke Model Implementation From ScratchFor the largest and most technologically advanced healthcare organizations, systems, hospitals, and provider organizations, creating and implementing bespoke GenAI model(s) is an attractive idea. That said, for most organizations today, it is not practical, as it depends on technical expertise and resourcing in addition to expanded governance at every stage of the AI model development. As the data sourcing, processing, model hosting, input tuning, and other technical elements are brought in-house, the risks associated with those steps are also shifted to the organization, which leads to the need for governance and technical controls.Potential cyber/compliance issues include:Subject Matter Expertise and Staffing RetentionProject complexity and integration of audit and security controlsComparing the Four Approaches Across Four ParametersCyber and Compliance Driven Implementation Approach Based on Risk Summary GridAs hospitals and health systems begin evaluating GenAI implementation approaches that fit their organization’s needs, it is helpful to reference the latest risks and vulnerabilities from cybersecurity industry leaders. With this best practice context, each organization can better choose the option that appropriately balances risk and benefits, with clearly defined governance around development, implementation, and usage.As we talk about the defense, it is worth validating the threat. The diagram below shows the dramatic increase in attacks focused on GenAI apps.From the Akamai blog “AI and LLM Bot Management Has Become a Business-Critical Issue: Do It Right” This shows the need to build out secure implementations.For each of the implementation approaches highlighted above, we call out the potential risks associated with a given stage of the development process based on the foundational industry standard is the Open Worldwide Application Security Project (formerly Open Web Application Security Project). The chart below includes the OWASP Top 10 for GenAI Applications and denoted those with their corresponding number based on the most recent 2025 list. The full list and definitions can be found here, including examples and additional context .Moving ForwardGenAI models provide clear benefits to healthcare organizations of all sizes. However, organizations must approach how they take advantage of these capabilities to avoid a crisis or safety issues. Finding the right tool and type of implementation for their business needs is imperative. We are now at the stage where AI models and implementations are moving quickly. In this dynamic time, it is important to take a step back, evaluate the business need, define governance, and proceed in a controlled, cautious manner. Healthcare organizations cannot afford to move fast and break things in clinical care settings.
About Author
