A patch can pass a benchmark's tests and still be wrong. I investigate how test adequacy affects measured agent performance, and how to make the evaluation harder to satisfy with incorrect behavior.
SWE-ABS / ICML 2026
What does a passing benchmark really prove?
Approach
SWE-ABS first uses program slicing to guide tests toward untested code, then uses plausible incorrect patches to generate adversarial tests that expose semantic blind spots.
Findings & contribution
The study strengthened tests for 50.2% of the 500 benchmark instances. The leading evaluated agent's success rate fell from 78.80% to 62.20%, and the previously top-ranked agent moved to fifth place. These figures describe the systems and benchmark evaluated in the paper.
SWE-ABS explained: a real weak test, the two-stage method, and the re-ranked leaderboard · 1:12
UTGenerator analyzes project code and dependencies to generate executable unit tests for real Python projects. UTBoost augments the benchmark test suites and re-evaluates agent patches against the stronger tests.
Findings & contribution
The study identified 36 instances with insufficient tests and 345 erroneous patches previously classified as passing. Re-evaluation changed 18 leaderboard positions on SWE-Bench Lite and 11 on Verified in the studied submissions.
UTBoost explained: a real insufficient test, generated tests, and the leaderboard changes · 1:06
For many programs, specifying the right answer is harder than running the program. My work explores relations between executions and between programs as a way to detect failures without enumerating correct outputs.
RT4CHART / arXiv 2026 (preprint)
Which claims in a RAG answer are supported by the retrieved context?
Approach
RT4CHART decomposes an answer into independently verifiable claims, checks them against the retrieved context, and assigns each claim an entailed, contradicted, or baseless label. Hierarchical verification maps these decisions back to answer spans with explicit supporting evidence.
Findings & contribution
On RAGTruth++ (408 samples), RT4CHART reports precision of 0.845, recall of 0.718, and F1 of 0.776. It also reports a span-level F1 of 0.475 on the re-annotated RAGTruth-Enhance dataset (2,675 samples). These results measure faithfulness to retrieved context, rather than open-domain factual correctness.
RT4CHART explained: a running example from the paper, claim-level local and global checks, and the results · 1:06
Inspired by inverse functions, the approach connects inputs and outputs through a forward program and a backward program. An auxiliary program can serve either role, allowing a relation between the original and transformed inputs to reveal inconsistencies.
Findings & contribution
The paper develops three testing modes and illustrates applications to algorithms, traditional software, and AI systems. The relation depends on the programs and task; it does not require every pair of programs to be exact inverses.
Example from the paper: test sin with its inverse arcsin
x0.5
P · auxiliaryarcsin(x) + 2kπ
yπ/6 + 2kπ
Q · under testsin(y)
x′0.5 = x ✓
If the sine implementation returns anything other than 0.5 for some k, the relation x = sin(arcsin(x) + 2kπ) fails and exposes the bug.
A conceptual view of Retromorphic Testing. The auxiliary program can play either role, which gives the paper's three testing modes.Retromorphic Testing explained: the forward–backward loop, three modes, and a translation example · 1:02
AI systems produce outputs that can be difficult to assess automatically. Controlled transformations and consistency relations provide practical ways to uncover mistakes in language and vision systems, and process-level checks show when an agent reaches the right outcome the wrong way.
CXBench / arXiv 2026 (preprint)
Does a correct outcome mean the agent followed the required process?
Approach
CXBench pairs each customer request with a tool-based environment and explicit process requirements across six regulated domains. Rule cards link cited regulatory obligations to observable trace evidence, and a hybrid grader combines deterministic checks for ordering and authority with an LLM judge for semantic requirements.
Findings & contribution
Across seven models and 120 tasks, 55.5% of the 840 retained executions were false passes: they reached the correct business outcome but failed a process check. These account for 80.1% of outcome-passing runs, and only 13.8% of executions passed both. Process-inclusive scoring also changed the observed model ordering.
CXBench explained: a false pass from the paper, rule cards, and outcome versus process scores · 1:04
Can controlled image changes reveal captioning errors?
Approach
ROME recursively removes and inpaints objects in an image, then compares the objects mentioned in captions before and after the transformation. This turns expected changes in image content into a metamorphic test oracle.
Findings & contribution
Across one commercial captioning API and four algorithms, the study reported 9,121 captioning errors from 226 seed images, with precision ranging from 86.47% to 92.17%.
ROME explained: object melting, two metamorphic relations, and the evaluation results · 0:57
Can consistency checks both detect and repair NER errors?
Approach
TIN compares entity predictions across related contexts and entities, using inconsistencies to report errors and guide automatic repairs.
Findings & contribution
In the evaluation of two models and the Azure and AWS APIs, 702 of 784 manually inspected reports were confirmed as errors. Automatic repair reduced errors by 26.8% to 50.6% across the evaluated systems.
TIN explained: a consistency check, a repair example, and the testing and repair results · 1:01
Can inserting an object reveal image captioning errors?
Approach
MetaIC builds an object corpus from existing images, inserts an object into a seed image with resizing and location tuning, and reports image pairs whose captions do not change as expected.
Findings & contribution
Testing one widely used captioning API and five captioning models with 1,000 seed images, MetaIC reported 16,825 captioning errors with precision from 84.9% to 98.4%. The same idea found 151 incorrect labels in the MS COCO Caption dataset.
MetaIC explained: object insertion, two metamorphic relations, and the evaluation results · 1:04
Reliable operational tools need evidence about accuracy, efficiency, and the impact of data preprocessing. I study these choices in log-based anomaly detection.
LightAD / ICSE 2024
When is a simpler model enough for log anomaly detection?
Approach
The study compares methods on HDFS, BGL, Spirit, Liberty, and Thunderbird. The accompanying LightAD toolkit combines preprocessing, deduplication, and automated model selection.
Findings & contribution
LightAD makes the trade-off between F1, training time, and inference time explicit in model selection. The comparison motivates evaluating classical baselines and preprocessing choices alongside deep learning models for log-based anomaly detection.
LightAD explained: F1 and training cost of simple and deep models, and automated model selection · 1:00
OmniFocus (2026 preprint) studies query-guided audio-visual token compression, estimating importance independently for each modality while preserving their alignment. Earlier work on mask-guided GANs (Neurocomputing 2021) uses text-body, outline, and shadow masks to guide scene text editing while preserving style and background.