Most AI vendor assessments are a cloud-security questionnaire with the word "AI" pasted in. They ask about encryption at rest and miss the questions that actually decide whether an agentic system is safe to run. These are the twenty-four that matter, what each one is really asking, and what a weak answer sounds like.
Who this is for
It is written to be useful from whichever side of the table you are sitting on, including the side we are usually on.
You have been handed an AI system to assess and your existing questionnaire does not reach it. Start here. The "what they are actually asking" line under each question is the one to read.
You built something that works and now it is stuck. Answer these in writing before your reviewer asks them. Arriving with the answers already written changes the shape of the conversation entirely.
You are evaluating vendors, including us. Ask all twenty-four. A vendor who answers them plainly is telling you something real; one who answers in the weak-answer column is telling you something too.
01 / 06
Where the software runs decides most of the rest. Settle this first, because every other answer changes depending on it.
Whether prompts containing your data cross your network boundary, and onto whose hardware. "In the cloud" is three different answers depending on whose cloud account, which region, and whether the model provider is a fourth party you have never assessed.
A weak answer sounds like
"It's hosted securely in the cloud with enterprise-grade encryption."
An exhaustive list of egress, not a summary. Telemetry, crash reports, usage analytics, and support bundles are all egress, and they are the ones vendors forget to mention because nobody thinks of them as data.
A weak answer sounds like
"Only anonymized metadata leaves your environment."
Whether bring-your-own-cloud is a deployment mode that exists today with customers running on it, or a roadmap item that will be built for you. Ask how many customers currently run that way. The number is the answer.
A weak answer sounds like
"We can absolutely discuss a private deployment for an enterprise agreement."
Whether there is a hard runtime dependency on the vendor — a licence check, a hosted control plane, a key service. This is a continuity question, but it is also the fastest way to find out how much of the system you would actually be operating.
A weak answer sounds like
"We maintain a business continuity plan and a source code escrow arrangement."
02 / 06
The vendor in front of you is rarely the only party in the inference path. These questions are about the ones behind them.
Both a contractual and a technical answer, and crucially one that covers the upstream model provider as well as the vendor. A vendor can honestly say they do not train on your data while sending it to a provider whose default terms say otherwise.
A weak answer sounds like
"We never train on customer data."
A named list with regions, treated as sub-processors under your data protection agreement. If the vendor routes across several providers for cost or availability, your data has more destinations than the contract names.
A weak answer sounds like
"We use industry-leading foundation models from major providers."
Most major providers retain inputs for a period for abuse monitoring even when they do not train on them, and that retention is often invisible to the vendor reselling the capability. Ask for the specific window and whether zero-retention is available on your tier.
A weak answer sounds like
"Nothing is retained — requests are processed and discarded."
Whether behaviour you validated can change underneath you without notice, and whether you can pin a version. A system qualified against one model and silently moved to another has not been qualified at all — which matters enormously in a regulated context.
A weak answer sounds like
"You always get the latest and most capable model automatically."
03 / 06
The most common structural flaw in enterprise AI is an agent that quietly holds more access than any of its users. This section is how you find it.
The single highest-value question in this document. If the agent holds its own credentials, every user of the system inherits the union of everything that account can reach — and your carefully built entitlement model has been replaced by one shared login.
A weak answer sounds like
"The agent has read-only access to the warehouse, so it's low risk."
Named protocols — SAML 2.0, OIDC — against your existing directory, with group and role claims actually consumed. "Supports SSO" can mean a separate user list that happens to accept a federated login, which means offboarding does not reach it.
A weak answer sounds like
"Yes, we support single sign-on."
Whether indexes, embeddings, caches, and conversation history outlive the entitlement that created them. Retrieval systems routinely copy data into a vector store that no longer consults the source system's permissions, which quietly defeats revocation.
A weak answer sounds like
"Permissions are checked at query time against the source."
Isolation across users and tenants, but specifically inside logs, traces, evaluation datasets, and support tooling. Conversation content is the most sensitive data the system holds and it is routinely the least protected, because it lives in observability rather than in the database.
A weak answer sounds like
"Our support team can access sessions when needed to troubleshoot."
04 / 06
A system that can only read is a search engine with opinions. The moment it can write, the risk profile changes completely — and this is where most assessments stop asking.
The literal list of write-capable tools it can invoke unattended, not a philosophy of autonomy. Ask for the tool manifest. If nobody can produce one, the honest answer is that nobody knows — which is itself the finding.
A weak answer sounds like
"There's always a human in the loop."
Whether anyone — an administrator, a support engineer, a future version — can turn it off to improve throughput. A control that can be disabled by configuration will eventually be disabled by configuration, usually during an incident when someone is in a hurry.
A weak answer sounds like
"Approval workflows are enabled by default for all write actions."
Whether approval is informed or ceremonial. A reviewer shown "the agent would like to update 12 records" will approve everything by the second week. A reviewer shown the exact diff, the source that justified it, and the count of affected rows is performing a control.
A weak answer sounds like
"The user gets a notification describing the proposed action."
Whether an approval is bounded. "Update the opportunity records" can mean twelve rows or sixty thousand, and an approval granted for one should not be reusable for the other. Ask whether the approval is pinned to the exact proposed change or to the intent.
A weak answer sounds like
"Actions are always scoped to the permissions of the requesting user."
05 / 06
The attack class with no complete defence. What matters is not whether a vendor claims to have solved it, but what happens when it works anyway.
Indirect prompt injection, which is the realistic version of this attack. Nobody types "ignore your instructions" into the chat box; it arrives inside a PDF, a support ticket, a résumé, or a supplier email that the agent was asked to summarise.
A weak answer sounds like
"We have guardrails that filter malicious prompts."
Whether the tool roster is scoped per turn or whether the whole registry is available on every call. If every tool is always reachable, a successful injection has the entire surface to work with rather than the three tools this task needed.
A weak answer sounds like
"The model intelligently selects the right tool for each request."
If the system writes and runs code — and most analytics agents do — the isolation boundary matters enormously. A shared-kernel container is a different risk class from hardware-level virtualization, and "sandboxed" is used for both.
A weak answer sounds like
"Code runs in an isolated sandbox environment."
Whether egress is denied by default with an allow-list, or permitted by default inside a private network. The second is the configuration that turns a contained code-execution bug into data exfiltration, and it is by far the more common one.
A weak answer sounds like
"It runs inside a private subnet with no public IP address."
06 / 06
Everything above is about preventing an incident. This section is about the day after one, which is when you discover whether any of it was reconstructable.
End to end: the request, every tool call and its arguments, every result, the model version, the approval and who granted it. "We log everything" usually means application logs with a thirty-day retention and no correlation between the two planes.
A weak answer sounds like
"All user activity is logged and retained per your policy."
Whether a source reference is produced by the backend from the actual tool-call record, or generated by the model as text. A model-generated citation can point at a document that was never retrieved, and it will look identical to a real one.
A weak answer sounds like
"The model always cites its sources for every claim."
Prompts and tool results contain whatever the user pasted in, which in practice includes customer records, credentials, and protected health information. Ask specifically whether prompt bodies reach the observability stack, and who administers that stack.
A weak answer sounds like
"All logs are encrypted at rest and in transit."
Vendor-side access, and specifically whether that access produces a record visible to you rather than only to them. An access log the customer cannot read is a control for the vendor's benefit, not yours.
A weak answer sounds like
"Access is limited to authorized personnel on a need-to-know basis."
Bonus
None of these appear on a standard assessment. All three tell you more about a vendor in one answer than the preceding twenty-four do.
A
Any vendor selling into regulated enterprises has failed one. A vendor who says they never have is telling you either that they have not been seriously reviewed, or that they are willing to say untrue things in a security assessment. Both are useful findings.
B
Every system has a component its own engineers are uneasy about. An answer names one and explains the mitigation. A non-answer insists there is nothing, which means either they have not thought about it or they are not going to tell you.
C
Fail-open or fail-closed, and who decided. A system that degrades to executing without its grounding checks, or to a weaker model nobody qualified, has a failure mode that only appears under load — which is exactly when nobody is watching.
Everything above is on this page in full and always will be — nothing here is gated. What the email gets you is the working copy: the twenty-four questions as an editable checklist you can put your own organization's name on, send to a vendor, and track answers in.
One email when it is sent, and one more if the list changes materially. Nothing else, and no sales sequence.
Sending… Got it — the editable checklist will reach your inbox within one business day. The questions above are yours either way. That did not send. Email info@axionalytics.com and it will be sent by hand — the questions above are unaffected.
If the system in question is yours and it is sitting in a review that will not clear, we answer all twenty-four in writing for it — with a threat model and a deployment topology — in five business days.