Publications
Delip Rao, Chris Callison-Burch. Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why. arXiv preprint, 2026.
Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels. Yet the same verdicts can support wildly varying agreement numbers, depending on seemingly minor choices: the judgment scale, the retained cases, the handling of abstentions and invalid outputs, and the pooling of verdicts across items and rubric criteria. The statistics that settle these choices are established, but in psychometrics, econometrics, and corpus annotation rather than in the evaluation practice that needs them. We treat the choices as a measurement protocol that fixes what the reported number estimates before any metric is computed, assemble the relevant results into a single source-attributed analysis, and apply it to three published LLM-judge evaluations. For non-degenerate binary verdicts, Pearson’s r, Spearman’s ρ, Kendall’s τb, the phi coefficient, and the Matthews correlation coefficient are exactly the same statistic, so reporting several repeats one number under different names. Cohen’s κ differs from them only through a marginal-mismatch factor in (0,1] and shares their asymptotic variance when judge and human assign the positive verdict equally often. Under exclusion, accuracy over all cases is pinned down only to a worst-case interval as wide as the uncovered fraction. On a rubric benchmark carrying per-criterion human labels, protocol choice alone moves reported accuracy from 0.551 to 0.899 and carries κ across zero, without altering a single verdict. We distill the analysis into a reporting checklist that makes agreement claims reconstructible and comparable.
@misc{rao_agreement_2026,
title = {Agreement {Metrics} for {LLM}-as-{Judge} {Evaluation}: {What} to {Report} and {Why}},
shorttitle = {Agreement {Metrics} for {LLM}-as-{Judge} {Evaluation}},
url = {http://arxiv.org/abs/2606.00093},
doi = {10.48550/arXiv.2606.00093},
abstract = {Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels. Yet the same verdicts can support wildly varying agreement numbers, depending on seemingly minor choices: the judgment scale, the retained cases, the handling of abstentions and invalid outputs, and the pooling of verdicts across items and rubric criteria. The statistics that settle these choices are established, but in psychometrics, econometrics, and corpus annotation rather than in the evaluation practice that needs them. We treat the choices as a measurement protocol that fixes what the reported number estimates before any metric is computed, assemble the relevant results into a single source-attributed analysis, and apply it to three published LLM-judge evaluations. For non-degenerate binary verdicts, Pearson's \$r\$, Spearman's \$ρ\$, Kendall's \$τ\_b\$, the phi coefficient, and the Matthews correlation coefficient are exactly the same statistic, so reporting several repeats one number under different names. Cohen's \$κ\$ differs from them only through a marginal-mismatch factor in \$(0,1]\$ and shares their asymptotic variance when judge and human assign the positive verdict equally often. Under exclusion, accuracy over all cases is pinned down only to a worst-case interval as wide as the uncovered fraction. On a rubric benchmark carrying per-criterion human labels, protocol choice alone moves reported accuracy from \$0.551\$ to \$0.899\$ and carries \$κ\$ across zero, without altering a single verdict. We distill the analysis into a reporting checklist that makes agreement claims reconstructible and comparable.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Rao, Delip and Callison-Burch, Chris},
month = jul,
year = {2026},
note = {arXiv:2606.00093 [cs.CL]},
keywords = {Computer Science - Computation and Language, Computer Science - Human-Computer Interaction, Physics - Data Analysis, Statistics and Probability},
}Delip Rao, Chris Callison-Burch. Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks. In The Third Annual Conference on Language Modeling (COLM), 2026.
Rubric-based LLM judges have become indispensable for evaluating and optimizing systems on non-verifiable tasks, where success cannot be reduced to exact programmatic checks. Yet the underlying judges remain vulnerable to position bias, stochastic inconsistency, criterion conflation, forced judgments under uncertainty, and model-dependent calibration. Rubrics structure evaluation but do not eliminate these failures; they add consequential choices about criterion design, scale types, weighting, aggregation, abstention, calibration, and reliability measurement. Despite extensive research across LLM evaluation, educational measurement, and psychometrics, the relevant methods remain scattered across papers and partial implementations. Researchers therefore pay a reinvention tax, repeatedly rebuilding evaluation machinery instead of accumulating knowledge on a common substrate. We introduce Autorubric, an open-source framework that makes rubric and judge choices explicit, reusable, and auditable. Through a unified API, it supports atomic evaluation of mixed criterion types alongside configurable bias mitigations, calibration, ensembling, abstention, and psychometric diagnostics. Evaluations spanning college chemistry grading, deep-research systems, and CHARM-100—a new mixed-criterion chatbot benchmark—reveal criterion-specific failures, systematic judge-family differences, and configuration effects that do not support a universal mitigation stack. We further demonstrate how per-criterion scores and explanations can support agent skill revision and reward modeling for reinforcement learning. By providing shared infrastructure from measurement through optimization, Autorubric gives the community a common basis for comparing methods, reproducing evaluation choices, and accumulating evidence across studies.
@misc{rao_autorubric_2026,
title = {Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks},
url = {http://arxiv.org/abs/2603.00077},
doi = {10.48550/arXiv.2603.00077},
abstract = {Rubric-based LLM judges have become indispensable for evaluating and optimizing systems on non-verifiable tasks, where success cannot be reduced to exact programmatic checks. Yet the underlying judges remain vulnerable to position bias, stochastic inconsistency, criterion conflation, forced judgments under uncertainty, and model-dependent calibration. Rubrics structure evaluation but do not eliminate these failures; they add consequential choices about criterion design, scale types, weighting, aggregation, abstention, calibration, and reliability measurement. Despite extensive research across LLM evaluation, educational measurement, and psychometrics, the relevant methods remain scattered across papers and partial implementations. Researchers therefore pay a reinvention tax, repeatedly rebuilding evaluation machinery instead of accumulating knowledge on a common substrate. We introduce Autorubric, an open-source framework that makes rubric and judge choices explicit, reusable, and auditable. Through a unified API, it supports atomic evaluation of mixed criterion types alongside configurable bias mitigations, calibration, ensembling, abstention, and psychometric diagnostics. Evaluations spanning college chemistry grading, deep-research systems, and CHARM-100---a new mixed-criterion chatbot benchmark---reveal criterion-specific failures, systematic judge-family differences, and configuration effects that do not support a universal mitigation stack. We further demonstrate how per-criterion scores and explanations can support agent skill revision and reward modeling for reinforcement learning. By providing shared infrastructure from measurement through optimization, Autorubric gives the community a common basis for comparing methods, reproducing evaluation choices, and accumulating evidence across studies.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Rao, Delip and Callison-Burch, Chris},
month = aug,
year = {2026},
note = {arXiv:2603.00077 [cs.CL]},
keywords = {Computer Science - Computation and Language, Computer Science - Artificial Intelligence},
}Delip Rao, Weiqiu You, Eric Wong, Chris Callison-Burch. NSF-SciFy: Mining the NSF Awards Database for Scientific Claims. In The 64th Annual Meeting of the Association for Computational Linguistics (ACL), 2026.
We introduce NSF-SciFy, a comprehensive dataset of scientific claims and investigation proposals extracted from National Science Foundation award abstracts. While previous scientific claim verification datasets have been limited in size and scope, NSF-SciFy represents a significant advance with 2.8 million claims from 400,000 abstracts spanning all science and mathematics disciplines. We present two focused subsets: NSF-SciFy-MatSci with 114,000 claims from materials science awards, and NSF-SciFy-20K with 135,000 claims across five NSF directorates. Using zero-shot prompting, we develop a scalable approach for joint extraction of scientific claims and investigation proposals. We demonstrate the dataset’s utility through three downstream tasks: non-technical abstract generation, claim extraction, and investigation proposal extraction. Fine-tuning language models on our dataset yields substantial improvements, with relative gains often exceeding 100%, particularly for claim and proposal extraction tasks. Our error analysis reveals that extracted claims exhibit high precision but lower recall, suggesting opportunities for further methodological refinement. NSF-SciFy enables new research directions in large-scale claim verification, scientific discovery tracking, and meta-scientific analysis.
@inproceedings{rao_nsf_scify_2026,
address = {San Diego, California, United States},
title = {{NSF}-{SciFy}: {Mining} the {NSF} {Awards} {Database} for {Scientific} {Claims}},
isbn = {9798891763906},
shorttitle = {{NSF}-{SciFy}},
url = {https://aclanthology.org/2026.acl-long.2118/},
doi = {10.18653/v1/2026.acl-long.2118},
abstract = {We introduce NSF-SciFy, a comprehensive dataset of scientific claims and investigation proposals extracted from National Science Foundation award abstracts. While previous scientific claim verification datasets have been limited in size and scope, NSF-SciFy represents a significant advance with 2.8 million claims from 400,000 abstracts spanning all science and mathematics disciplines. We present two focused subsets: NSF-SciFy-MatSci with 114,000 claims from materials science awards, and NSF-SciFy-20K with 135,000 claims across five NSF directorates. Using zero-shot prompting, we develop a scalable approach for joint extraction of scientific claims and investigation proposals. We demonstrate the dataset's utility through three downstream tasks: non-technical abstract generation, claim extraction, and investigation proposal extraction. Fine-tuning language models on our dataset yields substantial improvements, with relative gains often exceeding 100\%, particularly for claim and proposal extraction tasks. Our error analysis reveals that extracted claims exhibit high precision but lower recall, suggesting opportunities for further methodological refinement. NSF-SciFy enables new research directions in large-scale claim verification, scientific discovery tracking, and meta-scientific analysis.},
urldate = {2026-08-16},
booktitle = {Proceedings of the 64th {Annual} {Meeting} of the {Association} for {Computational} {Linguistics} ({Volume} 1: {Long} {Papers})},
publisher = {Association for Computational Linguistics},
author = {Rao, Delip and You, Weiqiu and Wong, Eric and Callison-Burch, Chris},
editor = {Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David},
month = jul,
year = {2026},
pages = {45679--45697},
}Muxin Liu, Delip Rao, Grace Kim, Chris Callison-Burch. When Verification Fails: How Compositionally Infeasible Claims Escape Rejection. In The Third Annual Conference on Language Modeling (COLM), 2026.
Scientific claim verification, the task of determining whether claims are entailed by scientific evidence, is fundamental to establishing discoveries in evidence while preventing misinformation. This process involves evaluating each asserted constraint against validated evidence. Under the Closed-World Assumption (CWA), a claim is accepted if and only if all asserted constraints are positively supported. We show that existing verification benchmarks cannot distinguish models enforcing this standard from models applying a simpler shortcut called salient-constraint checking, which applies CWA’s rejection criterion only to the most salient constraint and accepts when that constraint is supported. Because existing benchmarks construct infeasible claims by perturbing a single salient element they are insufficient at distinguishing between rigorous claim verification and simple salient-constraint reliance. To separate the two, we construct compositionally infeasible claims where the salient constraint is supported but a non-salient constraint is contradicted. Across model families and modalities, models that otherwise saturate existing benchmarks consistently over-accept these claims, confirming the prevalence of such shortcut reasoning. Via model context interventions, we show that different models and prompting strategies occupy distinct positions on a shared ROC curve, indicating that the gap between model families reflects differences in verification threshold rather than underlying reasoning ability, and that the compositional inference bottleneck is a structural property of current verification behavior that strategy guidance alone cannot overcome.
@misc{liu_when_2026,
title = {When {Verification} {Fails}: {How} {Compositionally} {Infeasible} {Claims} {Escape} {Rejection}},
shorttitle = {When {Verification} {Fails}},
url = {http://arxiv.org/abs/2604.10990},
doi = {10.48550/arXiv.2604.10990},
abstract = {Scientific claim verification, the task of determining whether claims are entailed by scientific evidence, is fundamental to establishing discoveries in evidence while preventing misinformation. This process involves evaluating each asserted constraint against validated evidence. Under the Closed-World Assumption (CWA), a claim is accepted if and only if all asserted constraints are positively supported. We show that existing verification benchmarks cannot distinguish models enforcing this standard from models applying a simpler shortcut called salient-constraint checking, which applies CWA's rejection criterion only to the most salient constraint and accepts when that constraint is supported. Because existing benchmarks construct infeasible claims by perturbing a single salient element they are insufficient at distinguishing between rigorous claim verification and simple salient-constraint reliance. To separate the two, we construct compositionally infeasible claims where the salient constraint is supported but a non-salient constraint is contradicted. Across model families and modalities, models that otherwise saturate existing benchmarks consistently over-accept these claims, confirming the prevalence of such shortcut reasoning. Via model context interventions, we show that different models and prompting strategies occupy distinct positions on a shared ROC curve, indicating that the gap between model families reflects differences in verification threshold rather than underlying reasoning ability, and that the compositional inference bottleneck is a structural property of current verification behavior that strategy guidance alone cannot overcome.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Liu, Muxin and Rao, Delip and Kim, Grace and Callison-Burch, Chris},
month = apr,
year = {2026},
note = {arXiv:2604.10990 [cs.CL]},
keywords = {Computer Science - Computation and Language, Computer Science - Artificial Intelligence},
}Delip Rao, Chris Callison-Burch. Bibtex citation hallucinations in scientific publishing agents: Evaluation and mitigation. In The Third Annual Conference on Language Modeling (COLM), 2026.
Large language models with web search are increasingly used in scientific publishing agents, yet they still produce BibTeX entries with pervasive field-level errors. Prior evaluations tested base models without search, which does not reflect current practice. We construct a benchmark of 931 papers across four scientific domains and three citation tiers — popular, low-citation, and recent post-cutoff — designed to disentangle parametric memory from search dependence, with version-aware ground truth accounting for multiple citable versions of the same paper. Three search-enabled frontier models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) generate BibTeX entries scored on nine fields and a six-way error taxonomy, producing ~23,000 field-level observations. Overall accuracy is 83.6%, but only 50.9% of entries are fully correct; accuracy drops 27.7pp from popular to recent papers, revealing heavy reliance on parametric memory even when search is available. Field-error co-occurrence analysis identifies two failure modes: wholesale entry substitution (identity fields fail together) and isolated field error. We evaluate clibib, an open-source tool for deterministic BibTeX retrieval from the Zotero Translation Server with CrossRef fallback, as a mitigation mechanism. In a two-stage integration where baseline entries are revised against authoritative records, accuracy rises +8.0pp to 91.5%, fully correct entries rise from 50.9% to 78.3%, and regression rate is only 0.8%. An ablation comparing single-stage and two-stage integration shows that separating search from revision yields larger gains and lower regression (0.8% vs. 4.8%), demonstrating that integration architecture matters independently of model capability. We release the benchmark, error taxonomy, and clibib tool to support evaluation and mitigation of citation hallucinations in LLM-based scientific writing.
@misc{rao_bibtex_2026,
title = {{BibTeX} {Citation} {Errors} in {Scientific} {Publishing} {Agents}: {Evaluation} and {Mitigation}},
shorttitle = {{BibTeX} {Citation} {Errors} in {Scientific} {Publishing} {Agents}},
url = {http://arxiv.org/abs/2604.03159},
doi = {10.48550/arXiv.2604.03159},
abstract = {Large language models with web search are increasingly used in scientific publishing agents, yet they produce BibTeX entries with pervasive field-level errors stemming from omission, partial corruption, substitution, and hallucination. We construct a benchmark of 931 papers across four domains and three citation tiers---popular, low-citation, and recent post-cutoff---with version-aware ground truth. Three search-enabled frontier models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) generate approximately 23,000 field-level observations. Overall accuracy is 83.6\%, but only 50.9\% of entries are fully correct; accuracy drops 27.7 pp from popular to recent papers, revealing heavy reliance on parametric memory even when search is available. Co-occurrence analysis identifies two failure modes: wholesale entry substitution and isolated field error. We present clibib, an open-source tool for deterministic BibTeX retrieval, as a mitigation mechanism. Two-stage integration raises accuracy to 91.5\% (+8.0 pp) and fully correct entries to 78.3\%, with a 0.8\% regression rate. Separating search from revision yields larger gains and lower regression than single-stage tool loops (0.8\% vs. 4.8\%), demonstrating that integration architecture matters independently of model/tool capability. We release clibib and an accompanying agent skill under the MIT License to improve citation accuracy in increasingly automated scientific workflows.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Rao, Delip and Callison-Burch, Chris},
month = aug,
year = {2026},
note = {arXiv:2604.03159 [cs.DL]},
keywords = {Computer Science - Digital Libraries, Computer Science - Computation and Language},
}Feijiang Han, Zelong Wang, Bowen Wang, Xinxin Liu, Skyler Cheung, Delip Rao, Chris Callison-Burch, Lyle Ungar. Latex2layout: High-fidelity, scalable document layout annotation pipeline for layout detection. In The 40th AAAI Conference on Artificial Intelligence (AAAI), 2026.
General-purpose Vision-Language Models (VLMs) are increasingly integral to modern AI systems for document understanding, yet their ability to perform fine-grained layout analysis remains severely underdeveloped. Overcoming this limitation requires large-scale, high-fidelity training datasets. However, current annotation methods that rely on parsing rendered PDFs are costly, error-prone, and difficult to scale. We propose a different paradigm: extracting ground-truth layout directly from the LaTeX compilation process rather than the final PDF. We present LaTeX2Layout, a generalizable procedural pipeline that recovers pixel-accurate bounding boxes and reading order from compiler traces. This enables the generation of a 140K-page dataset, including 120K programmatically generated synthetic variants that more than double the layout diversity of real-world data. Using this dataset, we fine-tune an efficient 3B-parameter VLM with an easy-to-hard curriculum that accelerates convergence. Our model achieves Kendall’s tau= 0.95 for reading order and mAP@ 50= 0.91 for element grounding, delivering nearly 200% relative improvement over strong zero-shot baselines such as GPT-4o and Claude-3.7.
@article{han_latex2layout_2026,
title = {{LaTeX2Layout}: {High}-{Fidelity}, {Scalable} {Document} {Layout} {Annotation} {Pipeline} for {Layout} {Detection}},
volume = {40},
copyright = {Copyright (c) 2026 Association for the Advancement of Artificial Intelligence},
issn = {2374-3468},
shorttitle = {{LaTeX2Layout}},
url = {https://ojs.aaai.org/index.php/AAAI/article/view/40349},
doi = {10.1609/aaai.v40i37.40349},
abstract = {General-purpose Vision-Language Models (VLMs) are increasingly integral to modern AI systems for document understanding, yet their ability to perform fine-grained layout analysis remains severely underdeveloped. Overcoming this limitation requires large-scale, high-fidelity training datasets. However, current annotation methods that rely on parsing rendered PDFs are costly, error-prone, and difficult to scale. We propose a different paradigm: extracting ground-truth layout directly from the LaTeX compilation process rather than the final PDF. We present LaTeX2Layout, a generalizable procedural pipeline that recovers pixel-accurate bounding boxes and reading order from compiler traces. This enables the generation of a 140K-page dataset, including 120K programmatically generated synthetic variants that more than double the layout diversity of real-world data. Using this dataset, we fine-tune an efficient 3B-parameter VLM with an easy-to-hard curriculum that accelerates convergence. Our model achieves Kendall's tau=0.95 for reading order and mAP@50=0.91 for element grounding, delivering nearly 200\% relative improvement over strong zero-shot baselines such as GPT-4o and Claude-3.7.},
language = {en},
number = {37},
urldate = {2026-08-16},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
author = {Han, Feijiang and Wang, Zelong and Wang, Bowen and Liu, Xinxin and Cheung, Skyler and Rao, Delip and Callison-Burch, Chris and Ungar, Lyle},
month = mar,
year = {2026},
pages = {30907--30915},
}Delip Rao, Eric Wong, Chris Callison-Burch. Detecting and correcting reference hallucinations in commercial llms and deep research agents. arXiv preprint, 2026.
Large language models and deep research agents supply citation URLs to support their claims, yet the reliability of these citations has not been systematically measured. We address six research questions about citation URL validity using 10 models and agents on DRBench (53,090 URLs) and 3 models on ExpertQA (168,021 URLs across 32 academic fields). We find that 3–13% of citation URLs are hallucinated — they have no record in the Wayback Machine and likely never existed — while 5–18% are non-resolving overall. Deep research agents generate substantially more citations per query than search-augmented LLMs but hallucinate URLs at higher rates. Domain effects are pronounced: non-resolving rates range from 5.4% (Business) to 11.4% (Theology), with per-model effects even larger. Decomposing failures reveals that some models fabricate every non-resolving URL, while others show substantial link-rot fractions indicating genuine retrieval. As a solution, we release urlhealth, an open-source tool for URL liveness checking and stale-vs-hallucinated classification using the Wayback Machine. In agentic self-correction experiments, models equipped with urlhealth reduce non-resolving citation URLs by 6–79x to under 1%, though effectiveness depends on the model’s tool-use competence. The tool and all data are publicly available. Our characterization findings, failure taxonomy, and open-source tooling establish that citation URL validity is both measurable at scale and correctable in practice.
@misc{rao_detecting_2026,
title = {Detecting and {Correcting} {Reference} {Hallucinations} in {Commercial} {LLMs} and {Deep} {Research} {Agents}},
url = {http://arxiv.org/abs/2604.03173},
doi = {10.48550/arXiv.2604.03173},
abstract = {Large language models and deep research agents supply citation URLs to support their claims, yet the reliability of these citations has not been systematically measured. We address six research questions about citation URL validity using 10 models and agents on DRBench (53,090 URLs) and 3 models on ExpertQA (168,021 URLs across 32 academic fields). We find that 3--13{\textbackslash}\% of citation URLs are hallucinated -- they have no record in the Wayback Machine and likely never existed -- while 5--18{\textbackslash}\% are non-resolving overall. Deep research agents generate substantially more citations per query than search-augmented LLMs but hallucinate URLs at higher rates. Domain effects are pronounced: non-resolving rates range from 5.4{\textbackslash}\% (Business) to 11.4{\textbackslash}\% (Theology), with per-model effects even larger. Decomposing failures reveals that some models fabricate every non-resolving URL, while others show substantial link-rot fractions indicating genuine retrieval. As a solution, we release urlhealth, an open-source tool for URL liveness checking and stale-vs-hallucinated classification using the Wayback Machine. In agentic self-correction experiments, models equipped with urlhealth reduce non-resolving citation URLs by \$6{\textbackslash}textrm\{--\}79{\textbackslash}times\$ to under 1{\textbackslash}\%, though effectiveness depends on the model's tool-use competence. The tool and all data are publicly available. Our characterization findings, failure taxonomy, and open-source tooling establish that citation URL validity is both measurable at scale and correctable in practice.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Rao, Delip and Wong, Eric and Callison-Burch, Chris},
month = apr,
year = {2026},
note = {arXiv:2604.03173 [cs.CL]},
keywords = {Computer Science - Computation and Language},
}Feijiang Han, Xiaodong Yu, Jianheng Tang, Delip Rao, Weihua Du, Lyle Ungar. ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.
Token-level attention tuning — a class of training-free methods including Post-hoc Attention Steering (PASTA) and Attention Calibration (ACT) — has emerged as a promising approach for improving frozen LLMs via interpretable interventions. However, these methods rely on auxiliary heuristics to identify important task-specific tokens, which can introduce bias and limit applicability when token importance is ambiguous or when optimized kernels make attention maps inaccessible. We propose a simpler alternative: intervening only on the initial token (e.g., in LLaMA). We theoretically show that adding lightweight biases to this token’s attention logits systematically shifts and reshapes downstream attention patterns — an effect amplified by its natural role as an attention sink. Empirically, we find that this tuning can improve LLM performance and better elicit pretrained knowledge, with stronger effects in early layers and distinct scaling preferences across attention heads. Building on these findings, we introduce ZeroTuning, a training-free method that improves LLM performance by applying head-specific attention adjustments to the initial token, requiring no parameter updates. We present two variants: a supervised mode that calibrates on validation examples, and an unsupervised mode that directly minimizes output entropy. ZeroTuning requires no KV-cache or decoding changes and is kernel-agnostic (works with SDPA and FlashAttention). It requires only four lines of modification to standard \texttt{LlamaAttention} code, achieves gains across 15 datasets, and outperforms prior, more complex methods. For example, on Llama-3.1-8B, it yields relative improvements of 19.9% on classification, 4.5% on question answering, and 2.1% on dialogue. ZeroTuning also works out of the box with quantized inference and maintains its improvements as context length increases. Our work provides a lightweight tool for inference-time improvement, advancing both optimization and interpretability. Our code and runnable demo are available at https://anonymous.4open.science/r/ZeroTuning.
@misc{han_zerotuning_2026,
title = {{ZeroTuning}: {Unlocking} the {Initial} {Token}'s {Power} to {Enhance} {Large} {Language} {Models} {Without} {Training}},
shorttitle = {{ZeroTuning}},
url = {http://arxiv.org/abs/2505.11739},
doi = {10.48550/arXiv.2505.11739},
abstract = {Token-level attention tuning, a class of training-free methods including Post-hoc Attention Steering (PASTA) and Attention Calibration (ACT), has emerged as a promising approach for improving frozen LLMs via interpretable interventions. However, these methods rely on auxiliary heuristics to identify important task-specific tokens, which can introduce bias and limit applicability when token importance is ambiguous or when optimized kernels make attention maps inaccessible. We propose a simpler alternative: intervening only on the initial token (e.g., BOS in LLaMA). We theoretically show that adding lightweight biases to this token's attention logits systematically shifts and reshapes downstream attention patterns - an effect amplified by its natural role as an attention sink. Empirically, we find that this tuning can improve LLM performance and better elicit pretrained knowledge, with stronger effects in early layers and distinct scaling preferences across attention heads. Building on these findings, we introduce ZeroTuning, a training-free method that improves LLM performance by applying head-specific attention adjustments to the initial token, requiring no parameter updates. We present two variants: a supervised mode that calibrates on validation examples, and an unsupervised mode that directly minimizes output entropy. ZeroTuning requires no KV-cache or decoding changes and is kernel-agnostic (works with SDPA and FlashAttention). It requires only four lines of modification to the standard LlamaAttention code, achieves gains across 15 datasets, and outperforms prior, more complex methods. For example, on Llama-3.1-8B, it yields relative improvements of 19.9\% on classification, 4.5\% on question answering, and 2.1\% on dialogue. ZeroTuning also works out of the box with quantized inference and maintains its improvements as context length increases.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Han, Feijiang and Yu, Xiaodong and Tang, Jianheng and Rao, Delip and Du, Weihua and Ungar, Lyle},
month = feb,
year = {2026},
note = {arXiv:2505.11739 [cs.CL]},
keywords = {Computer Science - Computation and Language, Computer Science - Artificial Intelligence},
}Delip Rao, Chris Callison-Burch. What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis. In The 31st International Conference on Applications of Natural Language to Information Systems (NLDB), 2026.
Despite rapid progress in claim verification, we lack a systematic understanding of what reasoning these benchmarks actually exercise. We generate structured reasoning traces for 24K claim-verification examples across 9 datasets using GPT-4o-mini and find that direct evidence extraction dominates, while multi-sentence synthesis and numerical reasoning are severely under-represented. A dataset-level breakdown reveals stark biases: some datasets almost exclusively test lexical matching, while others require information synthesis in roughly half of cases. Using a compact 1B-parameter reasoning verifier, we further characterize five error types and show that error profiles vary dramatically by domain – general-domain verification is dominated by lexical overlap bias, scientific verification by overcautiousness, and mathematical verification by arithmetic reasoning failures. Our findings suggest that high benchmark scores primarily reflect retrieval-plus-entailment ability. We outline recommendations for building more challenging evaluation suites that better test the reasoning capabilities verification systems need.
@incollection{cabrio_what_2027,
address = {Cham},
title = {What {Do} {Claim} {Verification} {Datasets} {Actually} {Test}? {A} {Reasoning} {Trace} {Analysis}},
volume = {16696},
isbn = {9783032295316 9783032295323},
shorttitle = {What {Do} {Claim} {Verification} {Datasets} {Actually} {Test}?},
url = {https://link.springer.com/10.1007/978-3-032-29532-3_4},
doi = {10.1007/978-3-032-29532-3_4},
language = {en},
urldate = {2026-08-16},
booktitle = {Natural {Language} {Processing} and {Information} {Systems}},
publisher = {Springer Nature Switzerland},
author = {Rao, Delip and Callison-Burch, Chris},
editor = {Cabrio, Elena and Monteiro, Eric},
year = {2027},
pages = {33--43},
}Delip Rao, Feijiang Han, Chris Callison-Burch. ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models. In The 31st International Conference on Applications of Natural Language to Information Systems (NLDB), 2026.
We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.
@incollection{cabrio_thinkncheck_2027,
address = {Cham},
title = {{ThinknCheck}: {Grounded} {Claim} {Verification} with {Compact}, {Reasoning}-{Driven}, and {Interpretable} {Models}},
volume = {16696},
isbn = {9783032295316 9783032295323},
shorttitle = {{ThinknCheck}},
url = {https://link.springer.com/10.1007/978-3-032-29532-3_2},
doi = {10.1007/978-3-032-29532-3_2},
language = {en},
urldate = {2026-08-16},
booktitle = {Natural {Language} {Processing} and {Information} {Systems}},
publisher = {Springer Nature Switzerland},
author = {Rao, Delip and Han, Feijiang and Callison-Burch, Chris},
editor = {Cabrio, Elena and Monteiro, Eric},
year = {2027},
pages = {18--32},
}Weiqiu You, Anton Xue, Shreya Havaldar, Delip Rao, Helen Jin, Chris Callison-Burch, Eric Wong. Probabilistic soundness guarantees in llm reasoning chains. In The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.
In reasoning chains generated by large language models (LLMs), initial errors often propagate and undermine the reliability of the final conclusion. Current LLM-based error detection methods often fail to detect propagated errors because earlier errors can corrupt judgments of downstream reasoning. To better detect such errors, we introduce Autoregressive Reasoning Entailment Stability (ARES), a probabilistic framework that evaluates each reasoning step based solely on previously-verified premises. This inductive method yields a nuanced score for each step and provides certified statistical guarantees of its soundness, rather than a brittle binary label. ARES achieves state-of-the-art performance across four benchmarks (72.1% Macro-F1,+ 8.2 points) and demonstrates superior robustness on very long synthetic reasoning chains, where it excels at detecting propagated errors (90.3% F1,+ 27.6 points).
@inproceedings{you_probabilistic_2025,
address = {Suzhou, China},
title = {Probabilistic {Soundness} {Guarantees} in {LLM} {Reasoning} {Chains}},
isbn = {9798891763326},
url = {https://aclanthology.org/2025.emnlp-main.382/},
doi = {10.18653/v1/2025.emnlp-main.382},
abstract = {In reasoning chains generated by large language models (LLMs), initial errors often propagate and undermine the reliability of the final conclusion. Current LLM-based error detection methods often fail to detect propagated errors because earlier errors can corrupt judgments of downstream reasoning. To better detect such errors, we introduce Autoregressive Reasoning Entailment Stability (ARES), a probabilistic framework that evaluates each reasoning step based solely on previously-verified premises. This inductive method yields a nuanced score for each step and provides certified statistical guarantees of its soundness, rather than a brittle binary label. ARES achieves state-of-the-art performance across four benchmarks (72.1\% Macro-F1, +8.2 points) and demonstrates superior robustness on very long synthetic reasoning chains, where it excels at detecting propagated errors (90.3\% F1, +27.6 points).},
urldate = {2026-08-16},
booktitle = {Proceedings of the 2025 {Conference} on {Empirical} {Methods} in {Natural} {Language} {Processing}},
publisher = {Association for Computational Linguistics},
author = {You, Weiqiu and Xue, Anton and Havaldar, Shreya and Rao, Delip and Jin, Helen and Callison-Burch, Chris and Wong, Eric},
editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
month = nov,
year = {2025},
pages = {7506--7525},
}Aditya M Kashyap, Delip Rao, Mary Regina Boland, Li Shen, Chris Callison-Burch. Predicting explainable dementia types with LLM-aided feature engineering. Bioinformatics, 2025.
The integration of Machine Learning and Artificial Intelligence (AI) into healthcare has immense potential due to the rapidly growing volume of clinical data. However, existing AI models, particularly Large Language Models (LLMs) like GPT-4, face significant challenges in terms of explainability and reliability, particularly in high-stakes domains like healthcare. This paper proposes a novel LLM-aided feature engineering approach that enhances interpretability by extracting clinically relevant features from the Oxford Textbook of Medicine. By converting clinical notes into concept vector representations and employing a linear classifier, our method achieved an accuracy of 0.72, outperforming a traditional n-gram Logistic Regression baseline (0.64) and the GPT-4 baseline (0.48), while focusing on high-level clinical features. We also explore using Text Embeddings to reduce the overall time and cost of our approach by 97%.
@article{kashyap_predicting_2025,
title = {Predicting explainable dementia types with {LLM}-aided feature engineering},
volume = {41},
copyright = {https://creativecommons.org/licenses/by/4.0/},
issn = {1367-4811},
url = {https://academic.oup.com/bioinformatics/article/doi/10.1093/bioinformatics/btaf156/8108827},
doi = {10.1093/bioinformatics/btaf156},
abstract = {Abstract
Motivation
The integration of Machine Learning and Artificial Intelligence (AI) into healthcare has immense potential due to the rapidly growing volume of clinical data. However, existing AI models, particularly Large Language Models (LLMs) like GPT-4, face significant challenges in terms of explainability and reliability, particularly in high-stakes domains like healthcare.
Results
This paper proposes a novel LLM-aided feature engineering approach that enhances interpretability by extracting clinically relevant features from the Oxford Textbook of Medicine. By converting clinical notes into concept vector representations and employing a linear classifier, our method achieved an accuracy of 0.72, outperforming a traditional n-gram Logistic Regression baseline (0.64) and the GPT-4 baseline (0.48), while focusing on high-level clinical features. We also explore using Text Embeddings to reduce the overall time and cost of our approach by 97\%.
Availability and implementation
All code relevant to this paper is available at: https://github.com/AdityaKashyap423/Dementia\_LLM\_Feature\_Engineering/tree/main.},
language = {en},
number = {4},
urldate = {2026-08-16},
journal = {Bioinformatics},
author = {Kashyap, Aditya M and Rao, Delip and Boland, Mary Regina and Shen, Li and Callison-Burch, Chris},
editor = {Wren, Jonathan},
month = mar,
year = {2025},
pages = {btaf156},
}Delip Rao, Jonathan Young, Thomas Dietterich, Chris Callison-Burch. WithdrarXiv: A Large-Scale Dataset for Retraction Study. arXiv preprint, 2024.
Now featured in Nature News! Retractions play a vital role in maintaining scientific integrity, yet systematic studies of retractions in computer science and other STEM fields remain scarce. We present WithdrarXiv, the first large-scale dataset of withdrawn papers from arXiv, containing over 14,000 papers and their associated retraction comments spanning the repository’s entire history through September 2024. Through careful analysis of author comments, we develop a comprehensive taxonomy of retraction reasons, identifying 10 distinct categories ranging from critical errors to policy violations. We demonstrate a simple yet highly accurate zero-shot automatic categorization of retraction reasons, achieving a weighted average F1-score of 0.96. Additionally, we release WithdrarXiv-SciFy, an enriched version including scripts for parsed full-text PDFs, specifically designed to enable research in scientific feasibility studies, claim verification, and automated theorem proving. These findings provide valuable insights for improving scientific quality control and automated verification systems. Finally, and most importantly, we discuss ethical issues and take a number of steps to implement responsible data release while fostering open science in this area.
@misc{rao_withdrarxiv_2024,
title = {{WithdrarXiv}: {A} {Large}-{Scale} {Dataset} for {Retraction} {Study}},
shorttitle = {{WithdrarXiv}},
url = {http://arxiv.org/abs/2412.03775},
doi = {10.48550/arXiv.2412.03775},
abstract = {Retractions play a vital role in maintaining scientific integrity, yet systematic studies of retractions in computer science and other STEM fields remain scarce. We present WithdrarXiv, the first large-scale dataset of withdrawn papers from arXiv, containing over 14,000 papers and their associated retraction comments spanning the repository's entire history through September 2024. Through careful analysis of author comments, we develop a comprehensive taxonomy of retraction reasons, identifying 10 distinct categories ranging from critical errors to policy violations. We demonstrate a simple yet highly accurate zero-shot automatic categorization of retraction reasons, achieving a weighted average F1-score of 0.96. Additionally, we release WithdrarXiv-SciFy, an enriched version including scripts for parsed full-text PDFs, specifically designed to enable research in scientific feasibility studies, claim verification, and automated theorem proving. These findings provide valuable insights for improving scientific quality control and automated verification systems. Finally, and most importantly, we discuss ethical issues and take a number of steps to implement responsible data release while fostering open science in this area.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Rao, Delip and Young, Jonathan and Dietterich, Thomas and Callison-Burch, Chris},
month = dec,
year = {2024},
note = {arXiv:2412.03775 [cs.CL]},
keywords = {Computer Science - Computation and Language, Computer Science - Digital Libraries, Computer Science - Machine Learning},
}Ajay Patel, Delip Rao, Ansh Kothary, Kathleen McKeown, Chris Callison-Burch. Learning Interpretable Style Embeddings via Prompting LLMs. arXiv preprint, 2023.
Style representation learning builds content-independent representations of author style in text. Stylometry, the analysis of style in text, is often performed by expert forensic linguists and no large dataset of stylometric annotations exists for training. Current style representation learning uses neural methods to disentangle style from content to create style vectors, however, these approaches result in uninterpretable representations, complicating their usage in downstream applications like authorship attribution where auditing and explainability is critical. In this work, we use prompting to perform stylometry on a large number of texts to create a synthetic dataset and train human-interpretable style representations we call LISA embeddings. We release our synthetic stylometry dataset and our interpretable style models as resources.
@misc{patel_learning_2023,
title = {Learning {Interpretable} {Style} {Embeddings} via {Prompting} {LLMs}},
url = {http://arxiv.org/abs/2305.12696},
doi = {10.48550/arXiv.2305.12696},
abstract = {Style representation learning builds content-independent representations of author style in text. Stylometry, the analysis of style in text, is often performed by expert forensic linguists and no large dataset of stylometric annotations exists for training. Current style representation learning uses neural methods to disentangle style from content to create style vectors, however, these approaches result in uninterpretable representations, complicating their usage in downstream applications like authorship attribution where auditing and explainability is critical. In this work, we use prompting to perform stylometry on a large number of texts to create a synthetic dataset and train human-interpretable style representations we call LISA embeddings. We release our synthetic stylometry dataset and our interpretable style models as resources.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Patel, Ajay and Rao, Delip and Kothary, Ansh and McKeown, Kathleen and Callison-Burch, Chris},
month = oct,
year = {2023},
note = {arXiv:2305.12696 [cs.CL]},
keywords = {Computer Science - Computation and Language},
}Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, Chris Callison-Burch. Faithful Chain-of-Thought Reasoning. arXiv preprint, 2023.
While Chain-of-Thought (CoT) prompting boosts Language Models’ (LM) performance on a gamut of complex reasoning tasks, the generated reasoning chain does not necessarily reflect how the model arrives at the answer (aka. faithfulness). We propose Faithful CoT, a faithful-by-construction framework that decomposes a reasoning task into two stages: Translation (Natural Language query symbolic reasoning chain) and Problem Solving (reasoning chain answer), using an LM and a deterministic solver respectively. We demonstrate the efficacy of our approach on 10 reasoning datasets from 4 diverse domains. It outperforms traditional CoT prompting on 9 out of the 10 datasets, with an average accuracy gain of 4.4 on Math Word Problems, 1.9 on Planning, 4.0 on Multi-hop Question Answering (QA), and 18.1 on Logical Inference, under greedy decoding. Together with self-consistency decoding, we achieve new state-of-the-art few-shot performance on 7 out of the 10 datasets, showing a strong synergy between faithfulness and accuracy.
@misc{lyu_faithful_2023,
title = {Faithful {Chain}-of-{Thought} {Reasoning}},
url = {http://arxiv.org/abs/2301.13379},
doi = {10.48550/arXiv.2301.13379},
abstract = {While Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of complex reasoning tasks, the generated reasoning chain does not necessarily reflect how the model arrives at the answer (aka. faithfulness). We propose Faithful CoT, a reasoning framework involving two stages: Translation (Natural Language query \${\textbackslash}rightarrow\$ symbolic reasoning chain) and Problem Solving (reasoning chain \${\textbackslash}rightarrow\$ answer), using an LM and a deterministic solver respectively. This guarantees that the reasoning chain provides a faithful explanation of the final answer. Aside from interpretability, Faithful CoT also improves empirical performance: it outperforms standard CoT on 9 of 10 benchmarks from 4 diverse domains, with a relative accuracy gain of 6.3\% on Math Word Problems (MWP), 3.4\% on Planning, 5.5\% on Multi-hop Question Answering (QA), and 21.4\% on Relational Inference. Furthermore, with GPT-4 and Codex, it sets the new state-of-the-art few-shot performance on 7 datasets (with 95.0+ accuracy on 6 of them), showing a strong synergy between faithfulness and accuracy.},
urldate = {2026-08-16},
publisher = {arXiv},
author = {Lyu, Qing and Havaldar, Shreya and Stein, Adam and Zhang, Li and Rao, Delip and Wong, Eric and Apidianaki, Marianna and Callison-Burch, Chris},
month = sep,
year = {2023},
note = {arXiv:2301.13379 [cs.CL]},
keywords = {Computer Science - Computation and Language},
}Nishant Subramani, Delip Rao. Learning Efficient Representations for Fake Speech Detection. In The 34th AAAI Conference on Artificial Intelligence (AAAI), 2020.
Synthetic speech or “fake speech” which matches personal vocal traits has become better and cheaper due to advances in deep learning-based speech synthesis and voice conversion approaches. This increased accessibility of synthetic speech systems and the growing misuse of them highlights the critical need to build countermeasures. Furthermore, new synthesis models evolve all the time and the efficacy of previously trained detection models on these unseen attack vectors is poor. In this paper, we focus on: 1) How can we build highly accurate, yet parameter and sample-efficient models for fake speech detection? 2) How can we rapidly adapt detection models to new sources of fake speech? We present four parameter-efficient convolutional architectures for fake speech detection with best detection F1 scores of around 97 points on a large dataset of fake and bonafide speech. We show how the fake speech detection task naturally lends itself to a novel multi-task problem further improving F1 scores for a mere 0.5% increase in model parameters. Our multi-task setting also helps in data-sparse situations, commonplace in adversarial settings. We investigate an alternative approach to the data-sparsity problem using transfer learning and show that it is possible to meet purely supervised detection performance for unseen attack vectors with as little as 6.25% of the training data. This is the first known application of transfer learning in adversarial settings for speech. Finally, we show how well our transfer learning approach adapts in an instance-efficient way to new attack vectors using the Real-Time Voice Cloning toolkit. We exceed the purely supervised detection performance (99.18 F1) with as little as 6.25% of the data.
@article{subramani_learning_2020,
title = {Learning {Efficient} {Representations} for {Fake} {Speech} {Detection}},
volume = {34},
copyright = {Copyright (c) 2020 Association for the Advancement of Artificial Intelligence},
issn = {2374-3468},
url = {https://ojs.aaai.org/index.php/AAAI/article/view/6044},
doi = {10.1609/aaai.v34i04.6044},
abstract = {Synthetic speech or “fake speech” which matches personal vocal traits has become better and cheaper due to advances in deep learning-based speech synthesis and voice conversion approaches. This increased accessibility of synthetic speech systems and the growing misuse of them highlights the critical need to build countermeasures. Furthermore, new synthesis models evolve all the time and the efficacy of previously trained detection models on these unseen attack vectors is poor. In this paper, we focus on: 1) How can we build highly accurate, yet parameter and sample-efficient models for fake speech detection? 2) How can we rapidly adapt detection models to new sources of fake speech? We present four parameter-efficient convolutional architectures for fake speech detection with best detection F1 scores of around 97 points on a large dataset of fake and bonafide speech. We show how the fake speech detection task naturally lends itself to a novel multi-task problem further improving F1 scores for a mere 0.5\% increase in model parameters. Our multi-task setting also helps in data-sparse situations, commonplace in adversarial settings. We investigate an alternative approach to the data-sparsity problem using transfer learning and show that it is possible to meet purely supervised detection performance for unseen attack vectors with as little as 6.25\% of the training data. This is the first known application of transfer learning in adversarial settings for speech. Finally, we show how well our transfer learning approach adapts in an instance-efficient way to new attack vectors using the Real-Time Voice Cloning toolkit. We exceed the purely supervised detection performance (99.18 F1) with as little as 6.25\% of the data.},
language = {en},
number = {04},
urldate = {2026-08-16},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
author = {Subramani, Nishant and Rao, Delip},
month = apr,
year = {2020},
pages = {5859--5866},
}Brian McMahan, Delip Rao. Listening to the World Improves Speech Command Recognition. In The 32nd AAAI Conference on Artificial Intelligence (AAAI), 2018.
We study transfer learning in convolutional network architectures applied to the task of recognizing audio, such as environmental sound events and speech commands. Our key finding is that not only is it possible to transfer representations from an unrelated task like environmental sound classification to a voice-focused task like speech command recognition, but also that doing so improves accuracies significantly. We also investigate the effect of increased model capacity for transfer learning audio, by first validating known results from the field of Computer Vision of achieving better accuracies with increasingly deeper networks on two audio datasets: UrbanSound8k and Google Speech Commands. Then we propose a simple multiscale input representation using dilated convolutions and show that it is able to aggregate larger contexts and increase classification performance. Further, the models trained using a combination of transfer learning and multiscale input representations need only 50% of the training data to achieve similar accuracies as a freshly trained model with 100% of the training data. Finally, we demonstrate a positive interaction effect for the multiscale input and transfer learning, making a case for the joint application of the two techniques.
@article{mcmahan_listening_2018,
title = {Listening to the {World} {Improves} {Speech} {Command} {Recognition}},
volume = {32},
copyright = {Copyright (c)},
issn = {2374-3468},
url = {https://ojs.aaai.org/index.php/AAAI/article/view/11284},
doi = {10.1609/aaai.v32i1.11284},
abstract = {We study transfer learning in convolutional network architectures applied to the task of recognizing audio, such as environmental sound events and speech commands. Our key finding is that not only is it possible to transfer representations from an unrelated task like environmental sound classification to a voice-focused task like speech command recognition, but also that doing so improves accuracies significantly. We also investigate the effect of increased model capacity for transfer learning audio, by first validating known results from the field of Computer Vision of achieving better accuracies with increasingly deeper networks on two audio datasets: UrbanSound8k and Google Speech Commands. Then we propose a simple multiscale input representation using dilated convolutions and show that it is able to aggregate larger contexts and increase classification performance. Further, the models trained using a combination of transfer learning and multiscale input representations need only 50\% of the training data to achieve similar accuracies as a freshly trained model with 100\% of the training data. Finally, we demonstrate a positive interaction effect for the multiscale input and transfer learning, making a case for the joint application of the two techniques.},
language = {en},
number = {1},
urldate = {2026-08-16},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
author = {McMahan, Brian and Rao, Delip},
month = apr,
year = {2018},
keywords = {deep learning, transfer learning, speech recognition},
}Delip Rao, Paul McNamee, Mark Dredze. Entity linking: Finding Extracted Entities in a Knowledge Base. In Multi-source, Multilingual Information Extraction and Summarization, Springer, 2013.
In the menagerie of tasks for information extraction, entity linking is a new beast that has drawn a lot of attention from NLP practitioners and researchers recently. Entity Linking, also referred to as record linkage or entity resolution, involves aligning a textual mention of a named entity to an appropriate entry in a knowledge base, which may or may not contain the entity. This has manifold applications ranging from linking patient health records to maintaining personal credit files, prevention of identity crimes, and supporting law enforcement. We discuss the key challenges present in this task and we present a high-performing system that links entities using max-margin ranking. We also summarize recent work in this area and describe several open research problems.
@incollection{poibeau_entity_2013,
address = {Berlin, Heidelberg},
title = {Entity {Linking}: {Finding} {Extracted} {Entities} in a {Knowledge} {Base}},
isbn = {9783642285684 9783642285691},
shorttitle = {Entity {Linking}},
url = {http://link.springer.com/10.1007/978-3-642-28569-1_5},
doi = {10.1007/978-3-642-28569-1_5},
urldate = {2026-08-16},
booktitle = {Multi-source, {Multilingual} {Information} {Extraction} and {Summarization}},
publisher = {Springer Berlin Heidelberg},
author = {Rao, Delip and McNamee, Paul and Dredze, Mark},
editor = {Poibeau, Thierry and Saggion, Horacio and Piskorski, Jakub and Yangarber, Roman},
year = {2013},
pages = {93--115},
}Delip Rao, David Yarowsky. Typed-Graph Models for Semi-supervised Learning of Name Ethnicity. In The 49th Annual Meeting of the Association for Computational Linguistics (ACL), 2011.
This paper presents an original approach to semi-supervised learning of personal name ethnicity from typed graphs of morphophonemic features and first/last-name co-occurrence statistics. We frame this as a general solution to an inference problem over typed graphs where the edges represent labeled relations be- tween features that are parameterized by the edge types. We propose a framework for parameter estimation on different constructions of typed graphs for this problem using a gradient-free optimization method based on grid search. Results on both in-domain and out-of-domain data show significant gains of over 30% accuracy improvement using the techniques presented in the paper.
@inproceedings{rao_typed_2011,
address = {Portland, Oregon, USA},
title = {Typed {Graph} {Models} for {Learning} {Latent} {Attributes} from {Names}},
url = {https://aclanthology.org/P11-2090/},
urldate = {2026-08-16},
booktitle = {Proceedings of the 49th {Annual} {Meeting} of the {Association} for {Computational} {Linguistics}: {Human} {Language} {Technologies}},
publisher = {Association for Computational Linguistics},
author = {Rao, Delip and Yarowsky, David},
editor = {Lin, Dekang and Matsumoto, Yuji and Mihalcea, Rada},
month = jun,
year = {2011},
pages = {514--518},
}Delip Rao, Michael Paul, Clay Fink, David Yarowsky, Timothy Oates, Glen Coppersmith. Hierarchical Bayesian Models for Latent Attribute Detection in Social Media. In The Fifth International AAAI Conference on Weblogs and Social Media (ICWSM), 2011.
We present several novel minimally-supervised models for detecting latent attributes of social media users, focusing on ethnicity and gender. Previous work on ethnicity detection has used coarse-grained, widely separated classes of ethnicity and assumed the existence of large amounts of training data, such as the US census, simplifying the problem. Instead, we examine content generated by users in addition to name morpho-phonemics to detect ethnicity and gender. Further, we address this problem in a challenging setting where the ethnicity classes are more fine-grained-ethnicity classes in Nigeria–and with very limited training data.
@article{rao_hierarchical_2011,
title = {Hierarchical {Bayesian} {Models} for {Latent} {Attribute} {Detection} in {Social} {Media}},
volume = {5},
copyright = {Copyright (c) 2021 Proceedings of the International AAAI Conference on Web and Social Media},
issn = {2334-0770},
url = {https://ojs.aaai.org/index.php/ICWSM/article/view/14197},
doi = {10.1609/icwsm.v5i1.14197},
abstract = {We present several novel minimally-supervised models for detecting latent attributes of social media users, with a focus on ethnicity and gender. Previouswork on ethnicity detection has used coarse-grained widely separated classes of ethnicity and assumed the existence of large amounts of training data such as the US census, simplifying the problem. Instead, we examine content generated by users in addition to name morpho-phonemics to detect ethnicity and gender. Further, weaddress this problem in a challenging setting where the ethnicity classes are more fine grained -- ethnicity classes in Nigeria -- and with very limited training data.},
language = {en},
number = {1},
urldate = {2026-08-16},
journal = {Proceedings of the International AAAI Conference on Web and Social Media},
author = {Rao, Delip and Paul, Michael and Fink, Clay and Yarowsky, David and Oates, Timothy and Coppersmith, Glen},
year = {2011},
pages = {598--601},
}Delip Rao, Paul McNamee, Mark Dredze. Streaming Cross-document Entity Coreference Resolution. In The 23rd International Conference on Computational Linguistics (COLING), 2010.
Previous research in cross-document entity coreference has generally been restricted to the offline scenario where the set of documents is provided in advance. As a consequence, the dominant approach is based on greedy agglomerative clustering techniques that utilize pairwise vector comparisons and thus require $O(n^2)$ space and time. In this paper, we explore identifying coreferent entity mentions across documents in high-volume streaming text, including methods for utilizing orthographic and contextual information. We test our methods using several corpora to quantitatively measure both the efficacy and scalability of our streaming approach. We show that our approach scales to at least an order of magnitude larger data than previously reported methods.
@inproceedings{rao_streaming_2010,
address = {Beijing, China},
title = {Streaming {Cross} {Document} {Entity} {Coreference} {Resolution}},
url = {https://aclanthology.org/C10-2121/},
urldate = {2026-08-16},
booktitle = {Coling 2010: {Posters}},
publisher = {Coling 2010 Organizing Committee},
author = {Rao, Delip and McNamee, Paul and Dredze, Mark},
editor = {Huang, Chu-Ren and Jurafsky, Dan},
month = aug,
year = {2010},
pages = {1050--1058},
}Mark Dredze, Paul McNamee, Delip Rao, Adam Gerber, Tim Finin. Entity Disambiguation for Knowledge Base Population. In The 23rd International Conference on Computational Linguistics (COLING), 2010.
The integration of facts derived from information extraction systems into existing knowledge bases requires a system to disambiguate entity mentions in the text. This is challenging due to issues such as non-uniform variations in entity names, mention ambiguity, and entities absent from a knowledge base. We present a state of the art system for entity disambiguation that not only addresses these challenges but also scales to knowledge bases with several million entries using very little resources. Further, our approach achieves performance of up to 95% on entities mentioned from newswire and 80% on a public test set that was designed to include challenging queries.
@inproceedings{dredze_entity_2010,
address = {Beijing, China},
title = {Entity {Disambiguation} for {Knowledge} {Base} {Population}},
url = {https://aclanthology.org/C10-1032/},
urldate = {2026-08-16},
booktitle = {Proceedings of the 23rd {International} {Conference} on {Computational} {Linguistics} ({Coling} 2010)},
publisher = {Coling 2010 Organizing Committee},
author = {Dredze, Mark and McNamee, Paul and Rao, Delip and Gerber, Adam and Finin, Tim},
editor = {Huang, Chu-Ren and Jurafsky, Dan},
month = aug,
year = {2010},
pages = {277--285},
}Delip Rao, David Yarowsky. Ranking and Semi-Supervised Classification on Large Scale Graphs Using Map Reduce. In The ACL Workshop on Graph-based Methods for Natural Language Processing (TextGraphs), 2009.
Label Propagation, a standard algorithm for semi-supervised classification, suffers from scalability issues involving memory and computation when used with large-scale graphs from real-world datasets. In this paper, we approach Label Propagation as a solution to a system of linear equations which can be implemented as a scalable parallel algorithm using the map-reduce framework. In addition to semi-supervised classification, this approach to Label Propagation allows us to adapt the algorithm to make it usable for ranking on graphs and derive the theoretical connection between Label Propagation and PageRank. We provide empirical evidence of that effect using two natural language tasks–lexical relatedness and polarity induction. The version of the Label Propagation algorithm presented here scales linearly in the data size with a constant main memory requirement, in contrast to the quadratic cost of both in traditional approaches.
@inproceedings{rao_ranking_2009,
address = {Suntec, Singapore},
title = {Ranking and {Semi}-supervised {Classification} on {Large} {Scale} {Graphs} {Using} {Map}-{Reduce}},
url = {https://aclanthology.org/W09-3209/},
urldate = {2026-08-16},
booktitle = {Proceedings of the 2009 {Workshop} on {Graph}-based {Methods} for {Natural} {Language} {Processing} ({TextGraphs}-4)},
publisher = {Association for Computational Linguistics},
author = {Rao, Delip and Yarowsky, David},
editor = {Choudhury, Monojit and Hassan, Samer and Mukherjee, Animesh and Muresan, Smaranda},
month = aug,
year = {2009},
pages = {58--65},
}Delip Rao, Deepak Ravichandran. Semi-supervised Polarity Lexicon Induction. In The 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2009.
We present an extensive study on the problem of detecting polarity of words. We consider the polarity of a word to be either positive or negative. For example, words such as good, beautiful, and wonderful are considered as positive words; whereas words such as bad, ugly, and sad are considered negative words. We treat polarity detection as a semi-supervised label propagation problem in a graph. In the graph, each node represents a word whose polarity is to be determined. Each weighted edge encodes a relation that exists between two words. Each node (word) can have two labels: positive or negative. We study this framework in two different resource availability scenarios using WordNet and OpenOffice thesaurus when WordNet is not available. We report our results on three different languages: English, French, and Hindi. Our results indicate that label propagation improves significantly over the baseline and other semi-supervised learning methods like Mincuts and Randomized Mincuts for this task.
@inproceedings{rao_semi_supervised_2009,
address = {Athens, Greece},
title = {Semi-{Supervised} {Polarity} {Lexicon} {Induction}},
url = {https://aclanthology.org/E09-1077/},
urldate = {2026-08-16},
booktitle = {Proceedings of the 12th {Conference} of the {European} {Chapter} of the {ACL} ({EACL} 2009)},
publisher = {Association for Computational Linguistics},
author = {Rao, Delip and Ravichandran, Deepak},
editor = {Lascarides, Alex and Gardent, Claire and Nivre, Joakim},
month = mar,
year = {2009},
pages = {675--682},
}Delip Rao, David Yarowsky, Chris Callison-Burch. Affinity Measures Based on the Graph Laplacian. In The COLING Workshop on Graph-based Algorithms for Natural Language Processing (TextGraphs), 2008.
Several language processing tasks can be inherently represented by a weighted graph where the weights are interpreted as a measure of relatedness between two vertices. Measuring similarity between arbitary pairs of vertices is essential in solving several language processing problems on these datasets. Random walk based measures perform better than other path based measures like shortest-path. We evaluate several random walk measures and propose a new measure based on commute time. We use the psuedo inverse of the Laplacian to derive estimates for commute times in graphs. Further, we show that this pseudo inverse based measure could be improved by discarding the least significant eigenvectors, corresponding to the noise in the graph construction process, using singular value decomposition.
@inproceedings{rao_affinity_2008,
address = {Manchester, UK},
title = {Affinity {Measures} {Based} on the {Graph} {Laplacian}},
url = {https://aclanthology.org/W08-2006/},
urldate = {2026-08-16},
booktitle = {Coling 2008: {Proceedings} of the 3rd {Textgraphs} workshop on {Graph}-based {Algorithms} for {Natural} {Language} {Processing}},
publisher = {Coling 2008 Organizing Committee},
author = {Rao, Delip and Yarowsky, David and Callison-Burch, Chris},
editor = {Matveeva, Irina and Biemann, Chris and Choudhury, Monojit and Diab, Mona},
month = aug,
year = {2008},
pages = {41--48},
}Delip Rao, Nikesh Garera, David Yarowsky. An Unsupervised Approach to Person Name Disambiguation using Web Snippets. In The Fourth International Workshop on Semantic Evaluations (SemEval), 2007.
This paper presents an approach to person name disambiguation using K-means clustering on rich-feature-enhanced document vectors, augmented with additional web extracted snippets surrounding the polysemous names to facilitate term bridging. This yields a significant F-measure improvement on the shared task training data set. The paper also illustrates the significant divergence between the properties of the training and test data in this shared task, substantially skewing results. Our system optimized on F0. 2 rather than F0. 5 would have achieved top performance in the shared task.
@inproceedings{rao_jhu1_2007,
address = {Prague, Czech Republic},
title = {{JHU1} : {An} {Unsupervised} {Approach} to {Person} {Name} {Disambiguation} using {Web} {Snippets}},
shorttitle = {{JHU1}},
url = {https://aclanthology.org/S07-1042/},
urldate = {2026-08-16},
booktitle = {Proceedings of the {Fourth} {International} {Workshop} on {Semantic} {Evaluations} ({SemEval}-2007)},
publisher = {Association for Computational Linguistics},
author = {Rao, Delip and Garera, Nikesh and Yarowsky, David},
editor = {Agirre, Eneko and Màrquez, Lluís and Wicentowski, Richard},
month = jun,
year = {2007},
pages = {199--202},
}Delip Rao, David Yarowsky. Part of Speech Tagging and Shallow Parsing of Indian Languages. In The IJCAI Workshop on Shallow Parsing for South Asian Languages (SPSAL), 2007.
This paper describes and evaluates shallow parsing of several Indian languages utilizing Conditional Random Field models. We show how performance can be substantially improved by several feature enhancements and improved modeling techniques, including expanding the chunk tag inventory, and separating punctuation from linguistic phrases. We also report results from part of speech tagging of Hindi, Bengali and Telugu using generative methods.
@inproceedings{rao2007indianpos,
title = {Part of Speech Tagging and Shallow Parsing of Indian Languages},
author = {Rao, Delip and Yarowsky, David},
booktitle = {Proceedings of the IJCAI Workshop on Shallow Parsing for South Asian Languages (SPSAL)},
address = {Hyderabad, India},
year = {2007},
}