https://arxiv.org/api/HePzlTFhUQJoTsxSJ9gxDQCmSjE 2026-09-28T21:57:00Z 15 44 0 http://arxiv.org/abs/2608.25241v2 2026-09-14T14:29:24Z

Coding agents increase development velocity but also technical debt. Prior work reports only average effects across adopters, hiding wide differences between teams. We introduce RAMP (Repository AI Maturity Profile), a four-level cumulative maturity model grounded in version-controlled artifacts that teams commit to configure AI tools. RAMP runs from behavioral rules and coding standards through named agent definitions to multi-agent orchestration, with observed practice concentrated in the first three levels. Across 441 repositories the levels behave as a cumulative scale, and independent human annotation reproduces RAMP's repository-level labels on 97% of a held-out sample. Adoption is cumulative, forward-only, and set-and-forget: 73.8% of artifacts are committed once and never modified. Re-estimating an existing agent-adoption panel within each stratum, agents accelerate development regardless of maturity (28-38% more commits), but quality diverges: among agent-first repositories, where the contrast is identified, those without committed AI configuration show roughly twice the increase in cognitive complexity (+53% versus +27%) and 1.7x the increase in static-analysis warnings. Because maturity is observational, correlated engineering discipline or model capability may explain part of the gap; we present these findings as hypothesis-generating and release RAMP as a reusable instrument.

2026-08-26T00:00:48Z Yegor Denisov-Blanch Shyam Agarwal Pavel Azaletskiy Hao He Rylan Schaeffer Brando Miranda Bogdan Vasilescu Sanmi Koyejo http://arxiv.org/abs/2607.05704v2 2026-08-26T22:56:41Z

Neural-network repositories contain executable models, recipes, input transformations, and measured accuracies. We study whether one same-family experiment can be curated as prompt guidance for LLM-based improvement of a low-performing target under equal generation and evaluation budgets. TuneNNGen extends NNGPT with a source-guided route and compares it with target-only generation on one CIFAR-10 target, two fixed source-selection rules, three code LLMs, and an additional SVHN target. Under the historical one-epoch search protocol, best-of-budget accuracy on the available evaluation split rises from 23.98% to 50.49% on CIFAR-10 and from 22.54% to 78.80% on SVHN. Selected five-epoch, three-seed means on train-derived validation splits retain gains of 40.94 points on CIFAR-10, 18.83 on Imagenette, and 7.27 on CIFAR-100. Direct-copy and negative-control analyses show that gains depend on source-target compatibility and LLM adaptation; stored source accuracy alone does not predict transferability.

2026-07-06T23:59:18Z 17 pages, 1 figure Kabir Dev Paul Baghel Radu Timofte Dmitry Ignatov http://arxiv.org/abs/2607.02854v1 2026-07-03T01:30:37Z

Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descriptions tend to be informal, making it difficult to determine whether a candidate BRT indeed fails for the reason in the issue. Prior work has attempted to tackle this problem via inference scaling, using large language models to generate many BRTs and patches, then using execution feedback to select and improve them. Unfortunately, this is expensive and the feedback is unreliable. This paper explores evolutionary programming for BRT generation to sharpen the feedback, while enhancing evolutionary programming to keep costs in check. Our new approach, EvoOtter, controls test execution costs via successive halving. Furthermore, it controls LLM costs via batched crossover for an entire generation in a single LLM call, as well as via rule-based code mutations, with a new fitness score tailored for BRTs. As a result, EvoOtter generates state-of-the-art quality BRTs at the fraction of the cost of prior inference-scaling approaches to this problem. More broadly, this paper points at how to efficiently and effectively combine evolutionary programming with large language models for software engineering.

2026-07-03T01:30:37Z Toufique Ahmed Jatin Ganhotra Avraham Shinnar Martin Hirzel http://arxiv.org/abs/2606.30383v1 2026-06-29T14:39:38Z

A rapidly growing class of LLM agents is multi-party: the agent acts for a principal (who briefs it, sends follow-ups, and receives results) while also conversing in a separate channel with a counterparty whose interests may diverge (negotiating with a vendor, screening inbound requests, or mediating between employees). Here "help whoever you are talking to" is the wrong objective. The agent must stay loyal to the principal it represents without over-refusing the principal's own cooperative asks. We study this multi-party loyalty problem and contribute a measurement instrument, two mechanisms, and a structural lesson. PrincipalBench is a 75-item multi-turn benchmark with leak probes, dual judges, and an integrity-audit gate. Across 13 frontier subjects it exposes a sharp split (<=20% vs. 53.6-75.3% harm) invisible to single-turn safety evaluations: a selective cluster that declines adversarial probes while still following the principal's legitimate requests, and an over-refusing cluster that refuses broadly. (M1) A prompt-time loyalty scaffold (a fixed system prompt of seven prioritized rules, open-coded from 50+ failure trajectories) holds Claude-Sonnet to 19.4% harm and all nine selective subjects to <=20%. (M2) A per-token-KL distillation recipe transfers a prompted Qwen3-32B teacher into 8B Qwen3 and Llama-3.1 students, the strongest open-weight recipe we measure. (Lesson) Both mechanisms only move along a common leak/over-refusal trade-off rather than crossing it: improving one axis costs the other, and the jointly favorable outcome stays out of reach.

2026-06-29T14:39:38Z Bojie Li Noah Shi http://arxiv.org/abs/2608.26124v1 2026-06-22T02:53:32Z

Pricing automation in large-scale tourism is challenging because travel orders are highly unstructured, while pricing policies are complex, rapidly evolving, and inherently open-ended. Traditional rule engines are brittle and costly to maintain, whereas unconstrained LLM agents lack the reliability and auditability required for financial decisions. We present a production-grade LLM-powered pricing system with a strict decision boundary: LLMs perform structured extraction and bounded policy/path selection, while all numeric pricing, including total-price computation, is executed deterministically. Policies are compiled into interpretable condition trees, enabling open-ended support for new clauses and evolving rules without code changes, while exposing auditable artifacts for human-in-the-loop control. Periodic fine-tuning on logged traces further improves tree induction and path matching. Deployed at a municipal state-owned tourism enterprise across 7 scenic sites and 12 business categories with 1,500+ operators and 1,000+ active policies, the system processed 3,960 orders in six months, reduced the order management team from 15-20 to 3, and cut per-order handling time from 10 minutes to <2 minutes.

2026-06-22T02:53:32Z Ziqiang Zhang Jing Ma Zilong Wang Jiayuan Chen Yi Qiao Yu He Wei Zhang Dai Cheng Xiaoyu Shen http://arxiv.org/abs/2606.04617v1 2026-06-03T08:53:36Z

Rules-as-Code promises more testable legal obligations, but it also changes what regulated firms can learn. Existing work mostly emphasizes implementation gains; the strategic gap is whether machine-readable rules make boundary search cheaper. I study that gap with a synthetic agent-based reinforcement-learning simulation that separates actual conduct near a legal threshold from proximity in the computable enforcement signal. Across 150 seed-level scenario runs, 378 common-random-number computability-sweep runs, 288 Latin-hypercube global-design runs, and a 2,880,000-row firm-period panel, computable static rules raise conduct boundary mass relative to ambiguous static rules (0.411 versus 0.367) and raise signal boundary mass more sharply (0.403 versus 0.281). Ordinary adaptive updates lower consumer harm (0.202 to 0.194) but do not reliably reduce boundary search. A budget-neutral anti-gaming design reduces conduct boundary mass by 0.032 and consumer harm by 0.025 relative to computable static rules. These are mechanism-oriented synthetic results, not estimates of real firm behavior in a jurisdiction or industry. The contribution is an estimand distinction, an inspectable ABM/RL mechanism, and a reproducible artifact showing that transparent behavioral assumptions are sufficient to generate gaming-like boundary dynamics without implying that computable regulation is inherently undesirable.

2026-06-03T08:53:36Z Includes synthetic simulation data, source code, figures, and reproducibility materials Xufeng He http://arxiv.org/abs/2605.29652v1 2026-05-28T09:16:21Z

Large language models (LLMs) are increasingly being used to generate health text from structured records such as wearable time series, biomarkers, vitals, and care-management logs. For recurring health outputs, fluency is not enough: systems must remain faithful to source data, ground explanatory claims in available evidence, follow stated policies, emit machine-readable outputs, and run cheaply enough for repeated use. We ask which responsibilities in structured health generation should be deterministic computation rather than runtime LLM prompting. We introduce Think Fast, Talk Smart, a sleep-health insight pipeline in which deterministic code performs recurring analysis before one bounded LLM writer call. Across 280 user-nights and six models, achieves lower numeric error, lower instruction-compliance error, and lower end-to-end cost than structured zero-shot and few-shot one-call baselines. Layer replacement reveals contract-specific failures: LLM comparison raises numeric error, LLM ranking degrades policy selection, LLM attribution increases unsupported causal language, and an LLM-generated writer interface reintroduces errors even after upstream facts are deterministic. The results support a broader design rule: let code own recurring analysis, and let LLMs express verified facts within bounded interfaces.

2026-05-28T09:16:21Z Kai-Chen Cheng Haejun Han David Q. Sun http://arxiv.org/abs/2605.19362v2 2026-05-20T17:49:17Z

Users often interpret and select agent skills through their SKILL markdown specifications. To protect users, existing audits mainly focus on malicious or unsafe skills. We study the complementary question of whether specifications help users form bounded expectations about what a skill consumes, produces, and covers. Across 878 cybersecurity skills, we used rule-based coding to measure textual cues for four comprehension anchors, namely operational basis, output contract, boundary disclosure, and example capability demonstration. Cues for operational basis were common, but only 19.0% of specifications exhibited cues for an example task, sample, or expected outcome, and only 2.3% exhibited cues for all four anchors. We further examined a small DNS/C2 telemetry subset (n$=$6) to illustrate why missing examples may matter. Examples appeared to make first local checks easier to construct, while no-example skills typically required helper code inspection to recover command arguments or output fields. We argue that agent-skill evaluation should treat specifications as user-facing capability disclosures, not merely as containers for executable instructions.

2026-05-19T04:50:42Z To appear at ACM CAIS Workshop Agent Skill 2026 Zikai Alex Wen http://arxiv.org/abs/2604.27643v1 2026-04-30T09:35:32Z

Integrated Circuit (IC) verification consumes nearly 70% of the IC development cycle, and recent research leverages Large Language Models (LLMs) to automatically generate testbenches and reduce verification overhead. However, LLMs have difficulty generating testbenches correctly. Unlike high-level programming languages, Hardware Description Languages (HDLs) are extremely rare in LLMs training data, leading LLMs to produce incorrect code. To overcome challenges when using LLMs to generate Universal Verification Methodology (UVM) testbenches and sequences, wepropose HAVEN (Hybrid Automated Verification ENgine) to prevent LLMs from writing HDL directly. For UVM testbench generation, HAVEN utilizes LLM agents to analyze design specifications to produce a structured architectural plan. The HAVEN Template Engine then combines with predefined and protocol-specific templates to generate all UVM components with correct bus-handshake timing. For UVM sequence generation, HAVEN introduces a Protocol-Aware Sequence Domain-Specific Language (DSL) that decomposes sequences into fine-grained step types. A set of predefined DSL patterns first establishes sequences that achieve a high coverage rate without LLM involvement. HAVEN continues to improve the coverage rate by iteratively leveraging LLM agents to analyze coverage gap reports and compose additional targeted DSL sequences. Unlike previous works, HAVEN is the first system that utilizes pre-defined, protocol-specific Jinja2 templates to generate all UVM components and UVM sequences using our proposed Protocol-Aware DSL and rule-based code generator. Our experimental results on 19 open-source IP designs spanning three interface protocols (Direct, Wishbone, AXI4-Lite) show that HAVEN achieves 100% compilation success, 90.6% code coverage, and 87.9% functional coverage on average, and is SOTA among LLM-assisted testbench generation systems.

2026-04-30T09:35:32Z 9 pages, 5 figures, 5 tables Chang-Chih Meng Yu-Ren Lu Guan-Yu Lin Tsung Tai Yeh Kai-Chiang Wu I-Chen Wu http://arxiv.org/abs/2604.00657v1 2026-04-01T09:04:01Z

Smart contracts are self-executing programs that manage financial transactions on blockchain networks. Developers commonly rely on third-party code libraries to improve both efficiency and security. However, improper use of these libraries can introduce hidden vulnerabilities that are difficult to detect, leading to significant financial losses. Existing automated tools struggle to identify such misuse because it often requires understanding the developer's intent rather than simply scanning for known code patterns. This paper presents LibScan, an automated detection framework that combines large language model (LLM)-based semantic reasoning with rule-based code analysis, identifying eight distinct categories of library misuse in smart contracts. To improve detection reliability, the framework incorporates an iterative self-correction mechanism that refines its analysis across multiple rounds, alongside a structured knowledge base derived from large-scale empirical studies of real-world misuse cases. Experiments conducted on 662 real-world smart contracts demonstrate that LibScan achieves an overall detection accuracy of 85.15\%, outperforming existing tools by a margin of over 16 percentage points. Ablation experiments further confirm that combining both analysis approaches yields substantially better results than either method used independently.

2026-04-01T09:04:01Z Yishun Wang Wenkai Li Xiaoqi Li Zongwei Li Lei Xie Yuqing Zhang http://arxiv.org/abs/2602.10480v4 2026-09-14T18:13:25Z

Large language models (LLMs) exhibit strong general-purpose reasoning capabilities, yet they frequently hallucinate when used as world models (WMs), where strict compliance with deterministic transition rules--particularly in corner cases--is essential. In contrast, Symbolic WMs provide logical consistency but lack semantic expressivity. To bridge this gap, we propose Neuro-Symbolic Synergy (NeSyS), a framework that integrates the probabilistic semantic priors of LLMs with executable symbolic rules to achieve both expressivity and robustness. NeSyS alternates training between the two models using trajectories inadequately explained by the other. Unlike rule-based prompting, the symbolic WM contributes candidate-level scores through log-linear reranking, without requiring the LLM to interpret rule text. Rule-guided sampling prioritizes transitions that are weakly covered by symbolic rules, using 35--60% of the training pairs while outperforming full-data supervised fine-tuning in five of six settings. Experiments on ScienceWorld, WebShop, and PlanCraft demonstrate consistent gains in WM prediction accuracy and data efficiency; one-step lookahead on open-ended WebShop also improves agent reward. Our models, rules, and code are available at https://github.com/tianyi-lab/NeSyS.

2026-02-11T03:36:18Z Camera-ready version accepted at COLM 2026 Hongyu Zhao Siyu Zhou Haolin Yang Zengyi Qin Tianyi Zhou http://arxiv.org/abs/2512.07109v1 2025-12-08T02:46:00Z

Responding to Hodel et al.'s (2024) call for a formal definition of task relatedness in re-arc, we present the first 9-category taxonomy of all 400 tasks, validated at 97.5% accuracy via rule-based code analysis. We prove the taxonomy's visual coherence by training a CNN on raw grid pixels (95.24% accuracy on S3, 36.25% overall, 3.3x chance), then apply the taxonomy diagnostically to the original ARC-AGI-2 test set. Our curriculum analysis reveals 35.3% of tasks exhibit low neural affinity for Transformers--a distributional bias mirroring ARC-AGI-2. To probe this misalignment, we fine-tuned a 1.7M-parameter Transformer across 302 tasks, revealing a profound Compositional Gap: 210 of 302 tasks (69.5%) achieve >80% cell accuracy (local patterns) but <10% grid accuracy (global synthesis). This provides direct evidence for a Neural Affinity Ceiling Effect, where performance is bounded by architectural suitability, not curriculum. Applying our framework to Li et al.'s independent ViTARC study (400 specialists, 1M examples each) confirms its predictive power: Very Low affinity tasks achieve 51.9% versus 77.7% for High affinity (p<0.001), with a task at 0% despite massive data. The taxonomy enables precise diagnosis: low-affinity tasks (A2) hit hard ceilings, while high-affinity tasks (C1) reach 99.8%. These findings indicate that progress requires hybrid architectures with affinity-aligned modules. We release our validated taxonomy,

2025-12-08T02:46:00Z 62 pages, 10 figures Miguel Ingram Arthur Joseph Merritt http://arxiv.org/abs/2510.06483v1 2025-10-07T21:46:26Z

Software applications often pose barriers for users with accessibility needs, e.g., visual impairments. Model-driven engineering (MDE), with its systematic nature of code derivation, offers systematic methods to integrate accessibility concerns into software development while reducing manual effort. This paper presents a systematic literature review on how MDE addresses accessibility for vision impairments. From 447 initially identified papers, 30 primary studies met the inclusion criteria. About two-thirds reference the Web Content Accessibility Guidelines (WCAG), yet their project-specific adaptions and end-user validations hinder wider adoption in MDE. The analyzed studies model user interface structures, interaction and navigation, user capabilities, requirements, and context information. However, only few specify concrete modeling techniques on how to incorporate accessibility needs or demonstrate fully functional systems. Insufficient details on MDE methods, i.e., transformation rules or code templates, hinder the reuse, generalizability, and reproducibility. Furthermore, limited involvement of affected users and limited developer expertise in accessibility contribute to weak empirical validation. Overall, the findings indicate that current MDE research insufficiently supports vision-related accessibility. Our paper concludes with a research agenda outlining how support for vision impairments can be more effectively embedded in MDE processes.

2025-10-07T21:46:26Z 41 pages Judith Michael Lukas Netz Bernhard Rumpe Ingo Müller John Grundy Shavindra Wickramathilaka Hourieh Khalajzadeh http://arxiv.org/abs/2509.21072v1 2025-09-25T12:23:49Z

Recent years, multimodal models have made remarkable strides and pave the way for intelligent browser use agents. However, when solving tasks on real world webpages in multi-turn, long-horizon trajectories, current agents still suffer from disordered action sequencing and excessive trial and error during execution. This paper introduces Recon-Act, a self-evolving multi-agent framework grounded in Reconnaissance-Action behavioral paradigm. The system comprises a Reconnaissance Team and an Action Team: the former conducts comparative analysis and tool generation, while the latter handles intent decomposition, tool orchestration, and execution. By contrasting the erroneous trajectories with successful ones, the Reconnaissance Team infers remedies, and abstracts them into a unified notion of generalized tools, either expressed as hints or as rule-based codes, and register to the tool archive in real time. The Action Team reinference the process empowered with these targeting tools, thus establishing a closed-loop training pipeline of data-tools-action-feedback. Following the 6 level implementation roadmap proposed in this work, we have currently reached Level 3 (with limited human-in-the-loop intervention). Leveraging generalized tools obtained through reconnaissance, Recon-Act substantially improves adaptability to unseen websites and solvability on long-horizon tasks, and achieves state-of-the-art performance on the challenging VisualWebArena dataset.

2025-09-25T12:23:49Z Kaiwen He Zhiwei Wang Chenyi Zhuang Jinjie Gu http://arxiv.org/abs/2509.10572v2 2025-09-21T02:54:05Z

Reliable data quality is crucial for downstream analysis of tabular datasets, yet rule-based validation often struggles with inefficiency, human intervention, and high computational costs. We present a three-stage framework that combines statistical inliner detection with LLM-driven rule and code generation. After filtering data samples through traditional clustering, we iteratively prompt LLMs to produce semantically valid quality rules and synthesize their executable validators through code-generating LLMs. To generate reliable quality rules, we aid LLMs with retrieval-augmented generation (RAG) by leveraging external knowledge sources and domain-specific few-shot examples. Robust guardrails ensure the accuracy and consistency of both rules and code snippets. Extensive evaluations on benchmark datasets confirm the effectiveness of our approach.

2025-09-11T14:17:42Z under review Ashlesha Akella Akshar Kaul Krishnasuri Narayanam Sameep Mehta