New Benchmark Reveals AI Agents Struggle With Long-Term Strategic Decisions
Researchers introduce Taste-Bench to measure decision quality, finding even top models fail on most strategic choices in long-running tasks.
Key points
- Taste-Bench tests how well AI agents make strategic choices during multi-step tasks, a gap in current benchmarks.
- The best model, GPT-5.6 Sol, selected the correct path on fewer than 60 percent of binary questions in the study.
- Extending the model’s thinking time did not improve its ability to identify better long-term trajectories.
Researchers from Microsoft and City University of Hong Kong have released a new benchmark called Taste-Bench that evaluates the strategic decision-making capabilities of AI agents during extended tasks, according to a report by The Decoder. The study finds that even the most advanced models struggle to distinguish high-quality paths from poor ones when those choices only reveal their value over time.
The benchmark specifically measures an agent's "taste," defined as its ability to make effective intermediate choices that lead to successful long-term outcomes. Existing evaluation methods typically only assess whether an agent completes a task, ignoring the quality of the steps taken along the way. Taste-Bench aims to fill this evaluation gap by analyzing the decision forks where agents diverge into different execution paths.
Context and Background
Evaluating long-term decisions is difficult because the consequences of a choice often do not appear immediately. A clean code implementation and a sloppy one might produce identical initial outputs and pass the same immediate tests, but the clean approach only demonstrates its value when future modifications become easier. Human grading for such nuances would be prohibitively expensive and require specialized domain expertise that does not scale, the researchers noted.
To address this, the researchers used hindsight to label decisions. They analyzed recorded trajectories of agent runs, which document every step and observation made during a task execution. Because agent systems often attempt the same task multiple times, these attempts naturally split into different directions, creating decision forks. The final result of each attempt reveals which path was superior, allowing the researchers to use the runs themselves as labeled comparisons to measure taste.
Methodology and Results
Taste-Bench consists of 502 questions, each presenting an original task, the trajectory up to a specific decision fork, and two possible subsequent directions. The first category of questions derives from two independent attempts at the same task that diverged, ensuring the agent never saw the incorrect path during its initial run. This tests the model's ability to choose between two unseen options based on context.
The second category comes from single runs where an agent corrected itself after taking a detour. In these cases, the initially discarded direction and the later successful path form the two candidates. This variant tests whether a model can identify a mistake earlier than the acting agent did. The raw data for these questions was sourced from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's public transcript release MALT.
What should you do?
Security and technology professionals should recognize that current AI agents are not yet reliable for autonomous, long-term strategic planning without human oversight. The study indicates that models cannot yet consistently predict which intermediate steps will yield better long-term results.
What should you do?
What to do and how to stay safe: Microsoft
- Review autonomous agent workflows to ensure human experts validate critical decision points, especially in long-running coding or research tasks.
- Monitor agent outputs for superficial success metrics that may hide underlying inefficiencies or poor structural choices made early in the process.
- Avoid relying on extended thinking times or additional compute to fix poor strategic judgment, as the study shows this does not improve decision quality.
- Wait for vendor updates or new benchmarks that specifically address long-term trajectory evaluation before trusting agents with high-stakes, multi-step projects.
Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
What is Taste-Bench and why was it created?
Taste-Bench is a new evaluation method created by Microsoft and City University of Hong Kong researchers to measure the quality of strategic decisions AI agents make during long-running tasks, rather than just final completion.
How did the best-performing model fare in the Taste-Bench study?
According to The Decoder, the best model, GPT-5.6 Sol, selected the correct strategic path on fewer than 60 percent of simple binary questions in the benchmark.
Does giving AI agents more time to think improve their strategic choices?
No, the researchers found that thinking longer did not help models make better long-term decisions or identify superior paths more accurately.
