Scoring Vendor Claims That AI Agents Work Like Human Staff Across EHRs and Payer Portals

Computer-using AI agents can perform healthcare administrative workflows without API integrations by navigating EHRs and payer portals through their screens, as staff do. That architecture addresses access while leaving completion uncertain. Evaluating claims that agents “work like human staff” therefore requires four tests, namely system coverage, workflow completeness, exception handling, and deployment reality.

Why "works like human staff" is a meaningful claim worth testing, not dismissing

Staff must gather information, reach the payer, submit the request, monitor its status, and handle whatever comes back. The burden crosses systems, often including payer portals with no API at all. Traditional RPA and API-only integrations still do not cover every system staff already use. An agent that can log in, read screens, fill forms, submit requests, monitor status, and escalate exceptions makes a specific operational claim. The lesson for administrators is to test that sequence before dismissing it because it sounds ambitious or accepting it because one demonstration looks convincing.

How do AI agents navigate EHRs and payer portals without API setup?

Two different architectures sit behind the same “works like human staff” language. API-based agents communicate through structured connections, including HL7 FHIR endpoints or EDI transactions. Those connections can be fast and reliable where supported, but their reach depends on what each EHR or payer accepts. Computer-use agents instead log into portals and interact with their interfaces. This is the route to operating in software without a supported API, including the payer portals involved in prior authorizations and eligibility checks.

The distinction determines what an administrator must evaluate. For an API-based agent, the question is whether the required connection exists and supports the actions the workflow needs. For a screen-operating agent, access through the interface must be followed by reliable navigation and completion. Production deployments commonly combine these approaches, using FHIR where available, portal navigation where it is not, and fax where necessary. A hybrid architecture makes the handoffs between channels part of the reliability test.

Epic’s Penny addresses revenue cycle work, Oracle Health has its Clinical AI Agent, athenahealth has agentic patient-communication tools, and eClinicalWorks has its AI API Workbench. These products show how much agents can now do inside the EHR, though their coverage does not extend as far across payer systems. Payer portal diversity remains the harder cross-system constraint. An API-only vendor may automate supported EHR-side steps effectively while still leaving staff responsible for every payer interaction without a supported connection.

What independent benchmarking reveals about end-to-end reliability

HealthAdminBench provides an empirical baseline for examining that distinction. Researchers at Stanford University, Stanford Health Care, and Kinetic Systems published the peer-reviewed benchmark at the ICLR 2026 AIWILD Workshop. It uses four realistic graphical environments, comprising an EHR, two payer portals, and a fax system. Expert-defined tasks cover prior authorization, appeals and denials management, and durable medical equipment order processing, with hundreds of verifiable subtasks. The scope tests coordination.

The benchmark’s best-performing agent, Claude Opus 4.6 CUA, achieved 36.3% full-task success. GPT-5.4 CUA achieved the highest subtask success rate, but individual-step performance and whole-workflow completion diverged sharply. Errors in handling documents and moving data between systems broke the sequence. These results provide a baseline limited to benchmark performance.

For administrators, the implication is precise. A demonstration of clinical data extraction, field entry, or decision reading establishes capability at that step. The agent must preserve the necessary information and complete the required actions through every handoff. The evaluation must therefore test the handoffs that a polished demonstration can leave out. The four dimensions below turn that gap into questions a practice can put to a vendor.

The four dimensions that separate genuine capability from marketing language

System coverage asks whether the agent can reach every system a staff member would use to do the proposed workflow. The concrete probe is a demonstration of the practice’s payer portals, with the vendor identifying which use APIs, screen navigation, or fax and explaining what happens when a payer is unsupported. Read access is not enough on its own. The demonstration should include write-back where the workflow requires it, including orders, coding, or prior-auth triggers. This dimension distinguishes a supported connection from the ability to perform the required work through that connection.

Workflow completeness asks where the vendor’s definition of “complete” ends. For prior authorization, the probe should establish whether the demonstrated process includes need-detection, submission building, submission routing, payer decision reading, appeal preparation, and status tracking, or stops at submission. The endpoint must be explicit before performance is scored. Otherwise, a vendor can describe a successful intermediate step using language that implies a finished workflow. Vendors that claim to automate complex appeals, off-formulary requests, and peer-to-peer reviews should show each of them, because people still do this work in many 2026 deployments.

Exception handling asks what happens when a step cannot be completed. The concrete probe is a live exception involving document handling or cross-system transfer, the handoffs identified by the benchmark. Some platforms use layered escalation through direct ePA rails, AI-generated structured fax, AI-powered phone outreach, and human review for complex cases. The vendor must show how the case reaches the appropriate next step, because a lower automation rate with a reliable human handoff is preferable to an incomplete submission that leaves staff unaware of the failure.

Deployment reality asks what the vendor means by “live.” Being capable of connecting to a system differs from operating reliably on it with the practice’s own payer mix. The probe is the actual time from signed contract to live transactions, including the credentials, access, integrations, and workflow changes required along the way. Staff effort belongs in that answer too. Before buying, an administrator needs to know what IT and administrative teams must do and whether the deployment needs resources the practice does not have.

How to weight the dimensions against what the practice actually needs to fix

The weights should follow the practice’s bottleneck. For a prior-auth-heavy orthopedic group, workflow completeness and payer coverage deserve the greatest weight, because automating submission building does little to resolve the backlog if the agent cannot reach the top three payers’ portals. For a primary care group with high denial and appeal rates, exception handling comes first. The decisive evidence is how the agent handles denial classification and appeal routing.

A multi-location group or PE/MSO portfolio faces a different constraint. With multiple EHRs and varied payer mixes, the same approach needs coverage across the existing estate to operate without API integrations or EHR replacements. For an initial deployment, deployment reality sets the timeline for when any return can begin. Workflow-specific KPIs should be agreed at the start, so the score reflects the problem being purchased away.

Compliance and PHI security as a pass/fail gate, not a scoring dimension

HIPAA applies to access to PHI regardless of whether an agent uses an API or operates a screen. Work in a payer portal that uses patient information therefore needs the same security assessment as other work with ePHI. The vendor should provide, at a minimum, SOC 2 Type II certification and a signed BAA that covers the agent configuration the practice will use. Their absence should disqualify a vendor before the four-dimensional evaluation begins.

Administrators should require documented answers about encryption in transit and at rest and the governance of PHI in API calls, logs, agent memory stores, temporary caches, and output channels. Screen-operating agents also warrant questions about local data stores and whether on-device semantic indexing or transcription can ingest regulated information. Prompt injection introduces a separate risk when clinical notes or web content redirect agent behavior, so vendors should explain detection and handling. BAA coverage can differ by feature, so assurances about the platform as a whole do not cover all the questions about the configuration the practice will use.

Where vendor claims currently hold up and where they still break down — the honest state of the field

Current capability is uneven across the workflow. Need detection, clinical data extraction, submission assembly, and approved-decision recording are among the stronger automated segments, with imaging and lab authorizations the most fully automated areas. Much of submission building can be automated even when the payer cannot accept electronic submission through the channel at hand. That leaves routing dependent on structured fax or another fallback. Complex specialty drugs and surgical services more often involve partial automation, while complex appeals, off-formulary requests, and peer-to-peer reviews still require human work in many deployments.

Allegheny Health Network illustrates what deployment at scale can look like. It processes hundreds of thousands of authorizations annually with a high first-pass approval rate using AI-driven automation spanning clinical data extraction, submission, and status tracking. That is a concrete operating scope. It supports the case that connected segments of prior authorization can be automated at substantial volume, without establishing that every exception or appeal is autonomous. The benchmark and the production example answer different questions, with one exposing general coordination failures and the other documenting an implemented workflow.

The CMS WISeR Model shows how to measure the results of the work beyond the number of automated actions. Its six-year, six-state Medicare pilot tracks AI for prior authorization across 13 service categories in outpatient, ASC, office, and home settings. The measures include timeliness, determination accuracy, favorable appeal decisions, and provider/supplier experience. Those outcomes reach beyond speed of submission. For a practice negotiating a contract, they connect the automation claim to whether work moves promptly, decisions are accurate, and the process serves the people using it.

Running the evaluation: what to ask, what to watch, and what a passing score looks like

The practical evaluation should begin with the compliance gate. Before showing the agent at work in the practice’s workflow, the vendor should provide the required documents and explain how the configuration the practice will use handles PHI. Once that gate is cleared, the demonstration should use a live workflow and the practice’s actual payer mix. The administrator’s focus should be the handoffs. Administrators should watch how documents move, how data passes between systems, and where the workflow ends more closely than how fast the agent acts on a screen.

The demonstration should then introduce exceptions deliberately. A payer portal that is down, a submission sent back as incomplete, and a denial that comes in an unexpected format each test whether the agent can send unfinished work to the right next step. The administrator should observe the escalation path rather than accept a description of what would happen. Deployment questions should follow that operational evidence, covering contract-to-live timing, required access, and prerequisites for IT or workflow changes. This separates an agent that can work in existing systems from a proposal whose value depends on a multi-month integration project.

The contract should translate demonstrated capability into workflow-specific commitments. First-pass approval rate, processing time per authorization, denial rate, and appeal conversion rate are relevant measures, with their use tied to the bottleneck being addressed. There is no universal passing percentage in this framework. A passing vendor documents the boundaries of autonomous operation, the points requiring human review, and the escalation path at each handoff, with a deployment burden the practice can support. A broad ROI promise cannot substitute for those commitments.

API-based automation can work well through supported connections, but on its own it cannot reach payer portals that have no supported API. Computer-use agents are best positioned to address that gap because they can operate across the EHR, payer portals, and other systems staff already touch without requiring API integrations, EHR replacements, or workflow overhauls. Their reach still needs proof of reliable completion.