Agentic AI platform evaluation starts with one request to the vendor: show me the record of a decision your agent prepared. If they can't show who acted, on what evidence, under which rules and who approved it, the rest of the demo doesn't matter much.
This checklist gives CIOs and CISOs a 12-question vendor question bank, what a good answer and a weak answer sound like, and the standard, regulation or incident behind each question. Use it in the RFP, in the demo and in the contract. Last updated 20 November 2026.
Ask these four live, in the demo. A weak answer to any one of them ends the evaluation.
Agentic AI platform evaluation starts with one request to see the record of a decision the vendor's agent prepared. Then ask the vendor to stop one agent live, show each agent's identity and permissions, and show which actions wait for a person. Treat those four as pass or fail, score the other eight, and run shadow mode on your own cases.
Agentic AI platform evaluation tests whether a vendor's agents can act inside your systems under your rules, with a record an auditor could follow later. A software review checks features. An agent review has to check behaviour, because the product takes actions you didn't script.
The market makes this harder. In June 2025 Gartner called the rebranding of assistants, robotic process automation and chatbots "agent washing", and estimated only about 130 of the thousands of agentic AI vendors are real. It also expects over 40% of agentic AI projects to be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls.
Buyers are moving faster than their controls. In a Q2 2026 Gartner survey of more than 2,000 CIOs and technology executives, 71% expected to deploy agentic AI within 12 months, but only 21% had agentic AI governance capabilities in place.
Regulators haven't waited for the market to settle. The SEC settled charges against two investment advisers in 2024 for false and misleading statements about their use of AI. In January 2025 its order against Presto Automation found the company had claimed its drive-thru product eliminated the need for human order taking, when it couldn't take orders on its own. The FTC said in 2024 that there is no AI exemption from the laws on the books.
The first four are gate questions, and a vendor that can't answer them well doesn't go further.
| # | Question for the vendor | Why ask it | A good answer | A weak answer |
|---|---|---|---|---|
| 1 | Show me the record of one decision your agent prepared. | The EU AI Act requires high-risk systems to log events automatically over their lifetime. | They open a real record with inputs, tool calls, model version, rules in force and approver, and replay it | A dashboard of success rates, or a chat transcript |
| 2 | Stop one agent mid-run in front of me. What does the record show afterwards? | The EU AI Act asks for a "stop" button or similar procedure, and the FSB's sound practices include testing kill switches. | One agent stops, its sub-agents stop, other agents keep running, and the stop is in the record | "We can switch off the integration" |
| 3 | What is each agent's identity, and what may it touch? | Six cyber agencies ask for each agent to be a distinct principal with its own keys or certificates. | Each agent has its own identity, a named owner, a written mandate and a permission list you can read | Agents run under a shared service account or a user's login |
| 4 | Which actions need a person's approval, and who sets that? | The same guidance says those decisions belong to designers or operators, not the agent. | You set approval points by action type and risk band, and irreversible actions always wait for a person | "The agent knows when to ask" |
| 5 | How do you keep each agent's access to the minimum? | OWASP asks for per-tool least-privilege profiles and short-lived, task-scoped tokens. | Per-tool scopes, credentials that expire after the task, production kept apart from test | One broad admin token |
| 6 | How did you test against prompt injection and agent hijacking? | NIST researchers raised hijacking success from 11% to 81% with new attacks. | Red-team results, which inputs count as untrusted, and what a hijacked agent still couldn't do | "Our model is safe" |
| 7 | Will it run on our own cases before we sign for production? | OMB M-25-22 tells US agencies to test proposed AI solutions, where practicable, when evaluating proposals. | Shadow mode on your data, with success measures agreed in advance | A curated demo only |
| 8 | What happens when the model underneath changes? | NIST's generative AI profile suggests vendor contracts address changes such as fine-tuning and drift. | Advance notice, the model version stamped on every record, a re-test and a rollback path | "We always use the latest model" |
| 9 | Who are your sub-processors and model providers, and where is our data processed? | GDPR Article 28 bars a processor from adding another processor without the controller's written authorisation. | A named list, regions, notice of changes and a right to object | "Our cloud partner handles that" |
| 10 | How do we leave, and what do we take with us? | DORA requires exit strategies with a transition period. | Data and decision records exported in an open format, with transition support and a tested exit | Export of chat logs only |
| 11 | How and when will you tell us about an incident? | The same NIST profile suggests asking for notice of serious incidents from third-party systems. | A notice time in the contract, who gets told, and help at a cost fixed in advance | "We'll keep you informed" |
| 12 | What independent assurance can you show, and what's in scope? | MAS guidelines point to independent certifications or assessments, not self-attestations. | A certificate or report with its scope statement and date | A logo on a slide |
A good answer always comes with something you can check. A weak answer asks you to trust the vendor.
Two requests sort most vendors in one meeting. Ask them to open the record of one decision and to stop one agent in front of you, before anyone talks price.
Score each question 0, 1 or 2. A 0 means no answer, or marketing. A 1 means the vendor described how it works. A 2 means they showed it live, on a real record, and gave you the document behind it.
Questions 1 to 4 are gates. Any 0 on a gate ends the evaluation, however well the vendor scores elsewhere. This scoring method is a FluxForce recommendation, not a published standard.
In its 2027 predictions, Forrester expects legal, security, compliance, procurement and technology leaders to demand evidence that a provider's AI can operate within their governance, audit and control requirements. More than half of AI purchases already involve four or more departments, so build the panel before the first demo.
The first figure shows the evaluation path. The vendor claim goes to the question bank first. The question bank sets what the live demonstration has to show. The live demonstration earns a run in shadow mode on your own cases. Shadow mode results feed the scorecard. The scorecard goes to the evaluation panel, and the panel makes the call.
Illustrative scenario, not a customer result.
An invented example, not a real customer. A 900-person insurer sends two agent platforms the 12 questions for claims triage.
IDC's analysts wrote in September 2026 that with agents, the record of what happened often doesn't exist at all.
A useful record has seven parts.
An agent passport is FluxForce's name for the first part, the record each agent carries of who it is, what it may see and do, which model it runs and who approved its last change. The FCA has seen firms keep a central register of agents showing which are live, who owns them and what they can access. A Cloud Security Alliance survey commissioned by a security vendor found 82% of organizations had AI agents running that they didn't know about.
The second figure shows how the record comes together. The agent passport goes into the decision record. The inputs and sources go into the decision record. Every tool call goes into the decision record. The model and prompt version go into the decision record. The approver's call goes into the decision record. The decision record is what an examiner replays months later.
Ask how long records are kept and who controls them. The EU AI Act asks deployers of high-risk systems to keep logs under their control for at least six months. There's no settled record format yet, since OpenTelemetry's conventions for tracing agents are still at Development status.
Ask for a live stop, not a slide. ISACA's 2026 poll found 56% of respondents didn't know how long it would take to halt an AI system after a security incident. The UK NCSC says that if an incident is detected, you should always be able to pull the plug and halt an AI agent that acts on its own.
Run the test in four steps.
The third figure shows the stop test. The kill switch stops the chosen agent mid-run. The chosen agent takes its sub-agents down with it. The kill switch leaves the other agents running. The chosen agent's last state is written to the decision record.
Permissions matter as much as the stop. In another vendor-commissioned CSA survey, 53% of organizations had AI agents exceed their intended permissions. In April 2026 a coding agent reportedly deleted a company's production database after finding an API token with blanket authority, including destructive operations. Its founder told Euronews there was no confirmation request before the action.
Buying an agent platform doesn't move accountability to the vendor. US banking agencies say using a third party doesn't diminish a bank's responsibility to meet its requirements, and DORA keeps EU financial entities fully responsible too.
Put these terms in the contract.
The PRA's updated SS2/21, effective from 18 March 2027, lists off-the-shelf machine learning models among third-party arrangements that still need adequate governance and controls. In the US, the Fed's SR 26-2 puts generative and agentic AI outside the revised model risk guidance, so your own risk management has to set the controls. That makes the contract, not the regulator's model guidance, the place where agent controls get written down.
This panel is a FluxForce recommendation. Each member owns the questions that match their job.
| Component | Responsibility | Boundary |
|---|---|---|
| CIO | Owns architecture fit, questions 7 and 8, and the final recommendation. | Doesn't sign without the CISO and compliance. |
| CISO | Owns questions 2, 3, 5 and 6 and runs the stop test. | Doesn't accept a shared service account. |
| Procurement and legal | Own questions 9 to 12 and the contract terms. | Don't accept a logo in place of a scope statement. |
| Compliance or risk | Owns question 1 and question 4, and reads the records. | Doesn't widen autonomy before shadow mode ends. |
| Business owner | Says whether the prepared work is right in shadow mode. | Doesn't score security questions. |
The evaluation panel makes the call, and nothing reaches production until shadow mode on your own cases is reviewed.
Send all 12 questions with the RFP, ask for the gate questions live in the demo, then run a shadow mode trial on your own cases.
Forrester found that 78% of buyers making purchases of $10 million or more run a trial first.
Automatic logs over the lifetime of high-risk systems; a stop button or similar procedure; deployer log retention of at least six months; Annex III duties from 2 December 2027.
EU law; high-risk duties only; check the consolidated text in force.
2026-11-20
Each agent as a distinct principal with its own keys; operators, not agents, decide when human approval is required.
Six national cyber agencies, May 2026; guidance, not law.
2026-11-20
Full responsibility stays with the firm; data locations with advance notice; audit rights; data return; exit with a transition period.
EU financial entities; applies from 17 January 2025.
2026-11-20
A 30-day plan. These steps are a FluxForce recommendation, not a regulatory requirement.
| Metric | Definition | Decision guardrail |
|---|---|---|
| Gate result | Whether the vendor scored 1 or 2 on each of questions 1 to 4. | Any 0 on a gate ends the evaluation. |
| Replay test | Share of sampled shadow mode decisions your compliance team could replay from the record alone. | A record that needs the vendor to explain it is a finding. |
FluxForce is the platform your departments run AI agents on. The agents investigate and prepare the work, a person makes the call, and every step leaves evidence an examiner can replay.
| Public stage | What happens |
|---|---|
| Take in | Agents connect beside the systems you already run, and nothing is migrated |
| Understand | The agent assembles the case into one view before anything is scored |
| Weigh | Your rules run first as hard checks, then the agent scores the case and recommends, with reasons |
| Prove | Every step goes into a tamper-evident record with its inputs, time, actor and the rules of that day |
| Report and learn | The agent drafts the report, a person approves it, and models change only after a person approves |
Governance controls and data protection apply on every step. Each agent carries an agent passport, works inside a mandate you set and has a kill switch.
The fourth figure shows the five stages. Take in passes the source records to understand. Understand passes one assembled view of the case to weigh. Weigh passes the checked and scored recommendation to prove. Prove passes the evidence record to report and learn.
Financial crime and compliance is the first app line on the platform: AI agents that investigate AML, sanctions, fraud and KYC alerts and prepare the case. Your analyst makes the call, with evidence an examiner can replay. You decide how much each agent does on its own, and every one has a kill switch.
For a CIO working through question 8, the closest fit is FluxForce AI Model Governance.
Fit and limits. Hold us to the same 12 questions, starting with a decision record and a live stop. For a department outside financial crime, ask which AI agents exist for it today and we'll tell you plainly. Our guide to AI agents in finance and where they fail covers where autonomy should stop inside a regulated firm.
Most agent demos show the agent working. Few show the record of a decision or what happens when you press stop, and that's where a real platform and a rebranded chatbot part ways.
Whoever you buy from, keep a person on every accountable call until shadow mode on your own cases has shown the agent deserves more.
Ask the vendor for evidence, not features. Use the 12 questions above, treat the decision record, the kill switch, agent identity and human approval points as pass or fail, and run the platform in shadow mode on your own cases before production.
Start with "show me the record of a decision your agent prepared" and "stop one agent in front of me". Then work through identity, approval points, injection testing, model changes, sub-processors, exit, incident notice and assurance. In September 2025 Gartner said it was too early to rely on a single vendor for an AI agent strategy.
No. ISO/IEC 42001 specifies requirements for an organization's AI management system, so a certificate tells you how the vendor runs AI, within a stated scope. It doesn't test the agent you're buying. Ask for the scope statement, then still run questions 1 to 4 live.
It can. Deployers of high-risk AI systems have their own duties, including keeping automatically generated logs under their control for at least six months. Under the AI Act as amended in 2026, the high-risk duties for Annex III systems apply from 2 December 2027.
An agent passport is FluxForce's term for the record each agent carries: who it is, what it may see and do, which model it runs, and who approved its last change. It's the first part of every decision record. Singapore's IMDA framework says an agent should have its own unique, cryptographically verifiable identity.