Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making
Evaluate AI models/agents on intraoperative perception, anesthesia decisions and decision updating.
Multimodal perception · Multi-step decisions · Anesthesiologist-aligned evaluation
The evaluation settings
Three complementary settings with different input representations, information access and temporal scopes.
L1
Read visible evidence
L2
Diagnosis at a single decision point
L3
Ongoing intraoperative diagnosis
Case analysis
Three decision turns during donor nephrectomy · DeepSeek-V4-Pro with the full agent framework.
Trajectory replay: later patient states and intervention outcomes come from existing case records; model recommendations do not change subsequent patient states.
MAP 67–69 mmHg · HR 49 bpm
The model described mild hypotension, proposed cautious fluids and conditional vasopressor use, and recommended observing the response.
The assessment noted that concurrent bradycardia was not explicitly addressed; Clinical Correctness, Evidence Grounding and Task Completeness had localized defects.
Safety label: safeMAP 101 mmHg
The model proposed observation and resuming remifentanil if MAP exceeded 105–110 mmHg.
The assessment judged the proposed threshold high, with a potential treatment delay; diagnosis and intervention retained localized correctness and grounding defects.
Safety label: minorMAP 78 mmHg · End-tidal CO₂ (EtCO₂) trend
The model proposed continued observation and ventilation adjustment if EtCO₂ continued to rise.
The assessment recognized management updating after pressure recovery, but noted that the diagnosis still described hypotension.
Safety label: safeEvidence adaptation and management coherence both received 2/2, while T1 still carried a minor safety concern.
A safe label means no safety concern was assigned in this assessment; it does not establish complete correctness or clinical usability.
Key findings
Reported results; no single score establishes clinical usability.
GPT-6-Astra’s TEE visual-grounding mIoU, the highest value in that column of Table 3. mIoU measures predicted–annotated region overlap; it is not clinical diagnostic accuracy.
GPT-6-Astra’s L3 Major/Critical Safety Error Rate. The denominator includes intervention responses with valid safety labels; this is not a patient adverse-event rate.
The same model’s L3 p95 latency: approximately 95% of valid timed turns were at or below this value. It sums model-request and tool-execution time per turn; it is neither an episode duration nor an operating-room response guarantee.
Published results
Explore the three-level evaluation setting. Scores are not directly comparable across levels. These published results include historical API-based models; new official submissions require locally executable weights.
Scroll horizontally for all metrics; click a column heading to sort. Arrows show the current sort direction.
How open-ended responses are assessed
AnesTRACE-Eval uses case facts and expert references, supported by retrieved passages from Miller’s Anesthesia, ASA guidance, pharmacological references and related literature. Patient-specific evidence available at the decision point remains primary.
For clinical teams
Assess the case sources, expert review, evaluator validation and evidence that has not yet been established.
Data come from VitalDB, MOVER, INSPIRE and EchoNet-TEE, with alignment, evidence-sufficiency, temporal and artifact checks before expert review. Selected retrospective cases do not represent every procedure or high-risk setting.
Models assist candidate generation; anesthesiologists review clinical relevance, correctness, decision-time evidence, anchor validity and evidence sufficiency. References relying on future information are removed.
Validation includes 100 L2 responses and 117 L3 decision-turn responses from 39 episodes. Three anesthesiologists share the annotation workload. Agreement is measured by correlation.
The domain-specific evaluator starts from Qwen3.5-9B, using approximately 7,000 expert-reviewed SFT examples and 4,000 expert-ranked DPO pairs. Training and benchmark cases overlap by design; responses from the validation generators are excluded from training. Agreement does not establish generalization to unseen clinical cases.
The results identify perception, response-quality, safety and decision-updating defects in the defined tasks. They do not establish improved patient outcomes from executing model recommendations or safe operating-room deployment. Unrepresented settings, unseen-case generalization and prospective workflow validation remain boundaries.
Model evaluation
Our inference code, agent framework, and evaluation tools are publicly available. To request evaluation on AnesTRACE-Bench, submit your model's Hugging Face repository or cloud-storage link through the model submission form. The team evaluates fixed-version model weights locally on the private benchmark. Benchmark data are not released due to license issues.
Clinical teams can review the case, protocol and evidence boundaries to assess the study design. The current public route supports model submissions; there is no public benchmark-data request process. Submissions are public GitHub Issues: do not upload case records, patient information, passwords or access tokens.
Reference
Paper
Code
AnesTRACE-Eval
@misc{huang2026anestrace,
title={AnesTRACE: Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making},
author={Ziwei Huang and Qi Gao and Zhe Ji and Yuanyuan Yao and Fengjiang Zhang and Min Yan and Zhongle Xie and Gang Chen},
year={2026},
eprint={2609.32740},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2609.32740}
}