Healthcare AI platform evaluation guide

How to use this guide

Send the questions in advance, ask for written answers, and score consistency between the written answer and what you hear on the call. Vagueness that survives a written request is the finding.

• Score each dimension out of five, and weight security, governance and integration highest — those are the ones that block deployment, not the ones that impress in a demo.
• Require evidence, not assertion. "We are HIPAA-compliant" is a claim. A BAA, a subprocessor list and a sample audit export are evidence.
• Ask the same questions of every vendor, including incumbents and internal build proposals.
• Record what was not answered. In our experience this column predicts implementation pain better than any score.

1. Security, HIPAA and business associate agreements

1. Will you sign a BAA? Does it cover every subprocessor that can access PHI, including model providers?
2. Publish your current subprocessor list. What notice do we get before it changes?
3. Is provider-side training on our submitted data contractually disabled? Show the clause.
4. How is data encrypted in transit and at rest, including vector stores, caches and logs?
5. Which certifications do you hold, what is their scope, and when were they last assessed?
6. Where does our data physically reside, and can we constrain residency?

Background on what each of these controls is doing: HIPAA-compliant generative AI. Our own answers: security, compliance and trust.

2. Data handling, retention and audit

1. What exactly is retained — prompts, retrieved context, outputs, embeddings, logs — and for how long?
2. Can you produce, for a named user and date range, every prompt, retrieval, model call and output?
3. How do you handle a patient-driven deletion request that touches derived data such as embeddings?
4. On termination, what is returned, what is destroyed, and on what timeline?
5. Are audit logs immutable, and who inside your organisation can read our data?

3. Governance and human oversight

1. Where is the human review step configured, and can it be removed by a non-engineer?
2. How is an agent's data access scoped, and is it independent of the builder's own permissions?
3. Can we see every agent in the estate, its owner, its data reach and its current model, in one view?
4. What change control governs the move from build to production?
5. How are model changes reviewed and recorded?

Our position: AI governance. Context on why scoping must be central: no-code AI agents.

4. Model flexibility and portability

1. Which providers do you support today, and can we use our own internal or self-hosted models?
2. Can we change the model behind a production agent without editing the agent? Demonstrate it.
3. Where do prompts and tool definitions live — in the platform, or written against one provider's API?
4. Do you provide normalised cost and token accounting across providers?
5. If we leave, what do we take with us: prompts, tool definitions, evaluation sets, logs?

What a credible answer looks like: multi-model AI for healthcare.

5. Retrieval quality and citation accuracy

1. Is retrieval filtered by the requesting user's permissions at query time, or is content filtered after generation?
2. How are clinical documents chunked, and how do you keep a dose attached to its indication?
3. Does every substantive claim in an answer carry a citation to a source passage with an effective date?
4. How does the system behave when the answer is not in the approved sources?
5. How do you evaluate retrieval quality separately from answer quality?
6. How often is the index refreshed, and how are permission changes propagated?

Detail on each: enterprise RAG for hospitals and health systems.

6. Agent capability and autonomy

1. What can an agent do without human review, and where is that boundary configured?
2. Can agents write to the EHR? If so, under what controls — and should they?
3. How are agent tool calls logged, and can a failing loop be bounded?
4. Show us three agents running in production at a comparable organisation, and what they do.
5. How many agents does your largest customer run, and who maintains them?

A grounded view of what is realistic: agentic AI for the healthcare sector.

7. Integration, identity and scalability

1. How do you integrate with Epic or our EHR, and what is read versus written?
2. Do you support SSO against our identity provider, with role mapping?
3. What does the first integration take in our engineering hours, honestly?
4. How does the tenth department differ from the first in effort?
5. What is your uptime commitment, and what happens to an in-workflow agent when a model provider degrades?

8. Evidence, references and total cost

1. Name production deployments at organisations comparable to ours, and let us speak to them unaccompanied.
2. How long from contract to first agent in production at those sites?
3. What did the customer's own team have to build?
4. Distinguish licence cost from model consumption cost from our internal engineering cost.
5. What has a customer asked you for that you could not deliver?

Frequently asked questions