How RAG works, in the order it actually runs
- Ingest. Approved sources — protocols, policies, formularies, tip sheets, payer rules — are collected with their metadata, ownership and effective dates.
- Chunk. Documents are split into retrievable passages. This step decides answer quality more than any other, and clinical documents punish naive splitting.
- Embed and index. Passages become vectors in a searchable store, alongside a keyword index. Access metadata travels with every chunk.
- Retrieve. The user's question is used to find candidate passages — filtered, always, by what that specific user is permitted to see.
- Rerank. Candidates are reordered by genuine relevance, which materially outperforms taking the top vector matches.
- Generate. The model answers using only the retrieved context, and is instructed to say when the context does not contain the answer.
- Cite. Every claim links back to the source passage, with its document, section and effective date.
Almost never at step 6. The recurring failure points are chunking that severs a dosage from its condition, an index that has drifted months out of date, and retrieval that ignores who is asking.
Permission-aware retrieval is not optional
The most consequential design decision in healthcare RAG is that filtering must happen during retrieval, not after generation. A system that retrieves broadly and then tries to redact has already put the content into the model's context and, usually, into a log.
- Access metadata on every chunk, mirroring the source system's permissions rather than a copy that drifts.
- Filter at query time using the requesting user's identity, so two clinicians asking the same question can legitimately receive different results.
- Re-check on refresh. When a source document's permissions change, the index must reflect it — stale access metadata is a live disclosure risk.
- Treat the vector store as PHI. Embeddings derived from clinical records inherit the encryption, retention and audit requirements of their source. See HIPAA-compliant generative AI.
Chunking clinical documents
Fixed-length chunking works acceptably on prose and badly on clinical material, where meaning is carried by structure: a dose belongs to a drug, which belongs to an indication, which belongs to a patient population.
| Document type | What retrieval must preserve |
|---|---|
| Clinical protocols | The condition, population and exclusion criteria that qualify each step. A step retrieved without its exclusions is actively dangerous. |
| Formulary and dosing | Drug, indication, dose, route and population as one unit, never split across chunks. |
| Payer policy | The payer, plan, effective date and the specific criterion. Policies supersede one another, so dates are part of the meaning. |
| Epic tip sheets | The workflow context and the version of the build the sheet describes. |
| Committee minutes and memos | Status — proposed, approved, superseded. Retrieving a rejected proposal as current guidance is a classic failure. |
In practice this means structure-aware chunking with generous overlap, metadata carried on every chunk, and a knowledge-graph layer for relationships that vector similarity cannot express — such as which policy supersedes which.
Reducing hallucination, and measuring that you have
RAG reduces hallucination; it does not eliminate it. A model given weak context will still produce a confident answer. The controls that matter are as much about measurement as about generation.
- Grounded generation. Instruct the model to answer only from retrieved context and to state plainly when the answer is not present.
- Mandatory citation. Every substantive claim carries a source link. An uncitable claim should not be displayed as an answer.
- Abstention as a success state. "This is not covered in your approved sources" is a correct answer, and should be measured as one rather than counted as a failure.
- Retrieval evaluation, separately from answer evaluation. Most quality problems are retrieval problems wearing a generation costume.
- Freshness checks. Surface each source's effective date so a reader can see when guidance is stale even if it was retrieved correctly.
RAG inside agents
In an agentic system, retrieval stops being a single step and becomes a tool the agent may call repeatedly, with queries it composes itself. That is more capable and harder to govern.
- Every retrieval call is logged with its query, its filters and what it returned — not just the final answer.
- The agent's retrieval scope is bounded at configuration time, so it cannot widen its own reach mid-task.
- Repeated retrieval is bounded, so a failing loop cannot pull large volumes of PHI into context.
- Citations survive multi-step reasoning: the final output still points at source passages, not at the agent's own intermediate summary.
See agentic AI for healthcare for how these bounds fit the broader autonomy question.
Frequently asked questions
What is enterprise RAG?
Enterprise RAG is retrieval-augmented generation applied to an organisation's own knowledge. Rather than relying on what a model learned in training, the system retrieves relevant passages from approved internal sources, passes them to the model as context, and returns an answer citing those sources. In a hospital that means answers grounded in the organisation's own protocols, formulary and payer rules.
Does RAG stop AI hallucination?
It reduces hallucination substantially but does not eliminate it. A model given poor or irrelevant retrieved context can still generate a confident, wrong answer. The controls that make RAG dependable are grounded generation prompts, mandatory citations, treating abstention as a correct outcome, and evaluating retrieval quality separately from answer quality.
Is a vector database secure enough for PHI?
A vector store holding embeddings derived from clinical records should be treated as a PHI repository: encrypted at rest, inside the same compliance boundary as the source data, subject to the same retention schedule, and filtered by the requesting user's permissions at query time rather than after generation.
Do we need a knowledge graph as well as a vector database?
For clinical knowledge, usually yes. Vector similarity finds passages that resemble the question, but it cannot express relationships such as which policy supersedes another, which protocol applies to which population, or which tip sheet matches the current EHR build. A graph layer carries those relationships; the vector index handles semantic recall.
How do we keep the index current?
Treat freshness as a monitored property, not a migration task. Sources are re-ingested on a schedule tied to how often they change, permission metadata is re-checked on every refresh, each chunk carries an effective date that is surfaced with the answer, and superseded documents are marked rather than silently deleted so a citation to old guidance still resolves.