Boxi YuEmail

Research

I develop testing and evaluation methods that uncover failures in software and AI, and examine what our tests actually establish.

All publicationsGoogle ScholarBibTeX

Evaluating coding agents

A patch can pass a benchmark's tests and still be wrong. I investigate how test adequacy affects measured agent performance, and how to make the evaluation harder to satisfy with incorrect behavior.

SWE-ABS / ICML 2026

What does a passing benchmark really prove?

Approach
SWE-ABS first uses program slicing to guide tests toward untested code, then uses plausible incorrect patches to generate adversarial tests that expose semantic blind spots.
Findings & contribution
The study strengthened tests for 50.2% of the 500 benchmark instances. The leading evaluated agent's success rate fell from 78.80% to 62.20%, and the previously top-ranked agent moved to fifth place. These figures describe the systems and benchmark evaluated in the paper.
PaperFull citation

UTBoost / ACL 2025

How rigorously do we evaluate coding agents?

Approach
UTGenerator analyzes project code and dependencies to generate executable unit tests for real Python projects. UTBoost augments the benchmark test suites and re-evaluates agent patches against the stronger tests.
Findings & contribution
The study identified 36 instances with insufficient tests and 345 erroneous patches previously classified as passing. Re-evaluation changed 18 leaderboard positions on SWE-Bench Lite and 11 on Verified in the studied submissions.
ACL paperCode & test artifactsVerified datasetFull citation

Constructing test oracles

For many programs, specifying the right answer is harder than running the program. My work explores relations between executions and between programs as a way to detect failures without enumerating correct outputs.

RT4CHART / arXiv 2026 (preprint)

Which claims in a RAG answer are supported by the retrieved context?

Approach
RT4CHART decomposes an answer into independently verifiable claims, checks them against the retrieved context, and assigns each claim an entailed, contradicted, or baseless label. Hierarchical verification maps these decisions back to answer spans with explicit supporting evidence.
Findings & contribution
On RAGTruth++ (408 samples), RT4CHART reports precision of 0.845, recall of 0.718, and F1 of 0.776. It also reports span-level F1 of 47.5% on the re-annotated RAGTruth-Enhance dataset (2,675 samples). These results measure faithfulness to retrieved context, rather than open-domain factual correctness.
PreprintFull citation

Retromorphic Testing / arXiv 2023 (preprint)

How do we test without a known answer?

Approach
Inspired by inverse functions, a forward program and a backward program connect inputs and outputs. An auxiliary program can serve either role, allowing a relation between the original and transformed inputs to reveal inconsistencies.
Findings & contribution
The paper develops three testing modes and illustrates applications to algorithms, traditional software, and AI systems. The relation depends on the programs and task; it does not require every pair of programs to be exact inverses.
  1. Original input
  2. Forward program
  3. Output
  4. Backward program
  5. Transformed input
Check the expected relation between the original and transformed inputs. A conceptual view of the dual-program architecture.
PreprintCodeFull citation

Testing AI systems

AI systems produce outputs that can be difficult to assess automatically. Controlled transformations and consistency relations provide practical ways to uncover mistakes in language and vision systems.

ROME / ISSTA 2023

Can controlled image changes reveal captioning errors?

Approach
ROME recursively removes and inpaints objects in an image, then compares the objects mentioned in captions before and after the transformation. This turns expected changes in image content into a metamorphic test oracle.
Findings & contribution
Across one commercial captioning API and four algorithms, the study reported 9,121 erroneous issues from 226 seed images, with precision ranging from 86.47% to 92.17%.
PaperCodeFull citation

TIN / ESEC/FSE 2023

Can consistency checks both detect and repair NER errors?

Approach
TIN compares entity predictions across related contexts and entities, using inconsistencies to report errors and guide automatic repairs.
Findings & contribution
In the evaluation of two models and the Azure and AWS APIs, 702 of 784 manually inspected reports were confirmed as errors. Automatic repair reduced errors by 26.8% to 50.6% across the evaluated systems.
PaperCodeFull citation

Empirical evaluation for AIOps

Reliable operational tools need evidence about accuracy, efficiency, and the impact of data preprocessing. I study these choices in log-based anomaly detection.

LightAD / ICSE 2024

When is a simpler model enough for log anomaly detection?

Approach
The study compares methods on HDFS, BGL, Spirit, Liberty, and Thunderbird. The accompanying LightAD toolkit combines preprocessing, deduplication, and automated model selection.
Findings & contribution
LightAD makes the trade-off between F1, training time, and inference time explicit in model selection. The comparison motivates evaluating classical baselines and preprocessing choices alongside deep learning models for log-based anomaly detection.
PaperCodeFull citation

Multimodal AI & visual computing

All publications

OmniFocus (2026 preprint) studies query-guided audio-visual token compression, estimating importance independently for each modality while preserving their alignment. Earlier work on mask-guided GANs (Neurocomputing 2021) uses text-body, outline, and shadow masks to guide scene text editing while preserving style and background.