Predicting Program Exit Code with LLMs and Programming Language Semantics

LLMs have become widely used in software engineering, demonstrating remarkable proficiency in tasks like code generation (Nijkamp et al., 2023; Guo et al., 2023; Lu et al., 2022), code translation (Yang et al., 2024a; Macedo et al., 2025), bug fixing (Hossain et al., 2024; Liu et al., 2024), and summarizing complex codebases (Makharev and Ivanov, 2025; Sun et al., 2025; Lomshakov et al., 2024). However, despite their practical success, it remains unclear to what extent these models truly comprehend the underlying semantics of the programs they process.

Much of an LLM’s apparent coding ability seems to come from sophisticated pattern matching and the memorization of vast amounts of training data, rather than an inherent ability to logically simulate program execution (Yang et al., 2024b; Meeus et al., 2026). For example, consider a scenario where an LLM is presented with a standard sorting algorithm. Because the model has previously encountered this exact structure many times, it can easily summarize its purpose, generate missing lines of code, or predict the final outcome (Štorek et al., 2026; Euraste et al., 2026). However, if we introduce a subtle change to the program, such as slightly modifying the loop bounds, obfuscating variable names, or redefining how the comparison operator works, and ask the model to predict the result, it frequently fails (Wang et al., 2024). Instead of systematically tracing the data flow step-by-step as an interpreter would, the model often falls back on its priors and simply guesses the standard sorted output (Malfa et al., 2024; Bernstein et al., 2026). Such a situation clearly reveals a gap between rote memorization and genuine, systematic semantic reasoning.

Recent work introduced PLSemanticsBench (Thimmaiah et al., 2026) and established a framework for evaluating LLMs’ semantic understanding through three tasks: predicting final variable states (PredState), identifying the formal semantic rules applied during execution (PredRule), and generating step-by-step execution traces (PredTrace) (Thimmaiah et al., 2026). Evaluations revealed that current LLMs struggle significantly across all three of these tasks. Because PLSemanticsBench presented a set of complex evaluations where models needed to articulate intermediate steps and output long execution traces, the models’ uniformly low performance leaves an open question regarding exactly where their capability threshold is. To determine whether these failures are in-part due to the natural multi-step complexity of the tasks presented in PLSemanticsBench or a fundamental inability to reason about semantics, we establish a more foundational baseline that separates trace generation from semantic judgment. Building on this premise, we present PrEx: a task that asks models, given a program and formal programming language semantics, to predict whether execution succeeds or halts with a semantic error—and, when it fails, which rule was violated. The required output is deliberately small with just a discrete executability verdict rather than a long trace but the evaluation is not: we pair each valid program with invalid counterparts produced by semantics-aware invalid transformations, vary program source and structural complexity across three splits, and probe rule-following under alternate semantic formalisms and shifts. This design lets us ask whether models systematically apply the supplied rules or instead continue to rely on memorized priors.

We evaluate open-source LLMs designed for coding tasks and enhanced reasoning using both non-chain-of-thought and chain-of-thought (CoT) reasoning. Our findings indicate that even on the simpler task (PrEx) of predicting program exit codes, LLMs continue to lean on priors learned during pre-training rather than systematically applying the supplied formal rules. Performance is especially poor under KeywordSwap and KeywordObf, where models must override familiar symbol meanings to follow the provided semantics. Accuracy also degrades significantly on the LLM-Translated and Fuzzer-Generated splits compared to Human-Written, indicating that semantic reasoning does not generalize well as structural complexity increases.

In traditional program interpretation, the semantic analyzer identifies semantic errors in the input program, typically following parsing (Aho et al., 2006; Appel, 1997). As a step towards our long term goal–evaluating whether LLMs can function as interpreters–in this paper, we tasked LLMs with detecting semantic errors.

We not only provide the model with a program it is expected to analyze but also supply the formal semantics of the language, including the inference rules that govern valid and invalid executions, thereby encouraging it to reason about semantic correctness. If an LLM identifies a program as semantically invalid, it is further instructed to specify the rule that formalizes the error, assessing its understanding of the programming language semantics. Consequently, we test not only if these models can identify a program as semantically invalid, but also if they lean on priors learned in their training data or on the semantics provided to them.

To illustrate the task, consider the example in Figure 1 of a C^∗ program which calculates the integer square root of the variable $n$. The program in Figure 1(a) is semantically valid and contains no rule violations, so an LLM successfully performing PrEx should classify it as valid, based on semantic reasoning over the inference rules rather than needing to simulate a full step-by-step execution. In contrast, Figure 1(b) shows an invalid variant of the same base program. Because line 8 of the invalid program contains a division-by-zero error, an LLM should successfully classify the program as invalid and accurately specify the violated rule.

This section describes our experimental setup for PrEx. We describe the C^∗ programming language (Section 3.1), the semantic formalisms used to specify program behavior ($\mathbb{S}$ and $\mathbb{K}$; Section 3.2), semantic shifts that test whether models follow provided rules or pre-training priors (KeywordSwap and KeywordObf; Section 3.3), and the full prompt given to models (Section 3.4). The C^∗ language, semantic formalisms, and semantics shifts are introduced in PLSemanticsBench (Thimmaiah et al., 2026), but we recap them here together with the full prompt used for PrEx.

The set of programs used in our evaluation are comprised of three splits: Human-Written, LLM-Translated, and Fuzzer-Generated. We use the same valid programs as PLSemanticsBench and extend their dataset with invalid programs, because PrEx requires both valid and invalid programs. We obtain these invalid programs by applying semantics-aware transformations to valid programs, so that each invalid counterpart is derived directly from a matched valid one with consistent style and complexity.

Table 4 shows program length across the three splits. Median lines of code (LOC) increases from Human-Written to LLM-Translated to Fuzzer-Generated. Even the shortest Fuzzer-Generated program is longer than the longest Human-Written program. We start with 491 valid programs⁸⁸ 8 The original PLSemanticsBench set had 492 valid programs, but we removed one Fuzzer-Generated program because its full PrEx prompt (C^∗ syntax, semantic rules, and program) requires $\sim$37K tokens and exceeds the 32K context window of our Qwen2.5-Coder models, leaving 491 semantically valid programs.. These programs are transformed to create 2455 semantically invalid programs. The transformation process uses an ANTLR-based parser-visitor to apply the five predefined transformations to each valid program, corresponding to the five semantic error rules under $\mathbb{S}$ and $\mathbb{K}$ formalizations. These transformations introduce specific semantic errors: 1) inserting a break outside a loop, 2) inserting a continue outside a loop, 3) division by zero, 4) modulo by zero, and 5) using an undeclared variable. Each invalid program is generated by applying one random transformation rule to a valid C^∗ program, ensuring each violates exactly one semantic error rule. For each valid program, we generate one corresponding semantically invalid program in each of the five error categories. Figure 4 illustrates these transformations on a shared valid program.

Table 5 shows the type of semantic errors introduced, their corresponding rules under $\mathbb{S}$ and $\mathbb{K}$ formalizations, and the number of programs in the dataset that violate each rule. The percentages are computed over the full set of 2946 programs.

We benchmark models under three settings by providing: the standard semantics and the unmodified program, KeywordSwap semantics and the transformed program, and KeywordObf semantics and the transformed program. Ground truths labels are derived from the transformation procedure. Valid programs are labeled ##success##, and each semantically invalid variant is labeled ##error## with the corresponding violated rule. We evaluate open-source LLMs designed for coding tasks and enhanced reasoning ability on PrEx, including models in the Qwen (Hui et al., 2024), DeepSeek (Guo et al., 2025), and Ministral (Liu et al., 2026) families. For non-reasoning models, we experiment with both CoT and non-CoT prompting. We run each model three times per configuration and report the mean accuracy across runs. Configurations are

Because reasoning models generate chain-of-thought traces by default, we evaluate them under one prompting condition only and do not run separate CoT and non-CoT variants.

Tables 6–8 summarize model performance on PrEx across the three dataset splits.

The quantitative results in Section 5 report rule-level accuracy aggregated over all programs and runs. To understand how models fail, we complement those aggregates with a qualitative pass over the three strongest PrEx models (DeepSeek-Qwen 32B, Qwen2.5-Coder 32B-CoT, and Ministral 3 14B-CoT). We only consider programs in the Human-Written split. We classify every misprediction into one of 4 failure modes:

• •

  False success: predicts ##success## for invalid program.

• •

  False error: predicts ##error## for valid program.

• •

  Wrong rule: correctly predicts invalidity but cites a different violated rule than the ground truth.

• •

  Malformed output: the model response does not contain a parseable <ans> tag.

This taxonomy uses the same rule-level criterion as the main accuracy tables: a prediction counts as correct only when both executability and the violated rule (on invalid programs) match the ground truth.

For each of the 6 PrEx configurations ($\mathbb{K}$/$\mathbb{S}$ $\times$ Standard/KeywordSwap/KeywordObf), we select the 2 smallest (by line count) non-empty failing programs for each top model and inspect the full model responses. The goal is not to cherry-pick hard cases, but to examine minimal counterexamples of programs where even short, explicit code still triggers systematic errors. Table 9 shows 2 of the shortest Human-Written failures under $\mathbb{S}$-semantics (5 lines each). Both programs are semantically invalid for a single, localized reason; neither requires deep control-flow reasoning.

In the first example, continue appears at the top level with an empty control stack, violating Rule 76. DeepSeek-Qwen 32B and Ministral 3 14B-CoT identify this correctly, but Qwen2.5-Coder 32B-CoT predicts invalid while citing Rule 73 (break-outside-loop) instead. This matches the per-error-type pattern in Figure 5, where continue-outside-loop and break-outside-loop identification degrades more than arithmetic errors.

In the second example, ans is initialized to 0 and then used as a modulo divisor. All three models predict invalid and the correct error type (modulo-by-zero), but DeepSeek-Qwen 32B cites Rule 24 rather than Rule 23 (Rule 24 is an unrelated unary expression rule). The model therefore applies the right semantic reasoning (modulo-by-zero) but maps it to a neighboring rule in the $\mathbb{S}$ specification. Such off-by-one rule citations still count as incorrect under our metric even when the natural-language explanation is essentially right.

We introduced Program Executability Prediction (PrEx), a task that asks models, when given a program and formal programming language semantics, to predict whether execution succeeds or halts on a semantic error and, when it fails, which rule was violated. Through PrEx, we can evaluate whether LLMs apply explicitly provided programming-language semantics or fall back on pre-training priors. Building on valid C^∗ programs from PLSemanticsBench, we extended the benchmark with invalid programs produced by five semantics-aware invalid transformations, and evaluated open-source coding LLMs under $\mathbb{S}$ and $\mathbb{K}$ formalisms, KeywordSwap and KeywordObf semantic shifts, and three program data splits (Human-Written, LLM-Translated, and Fuzzer-Generated). We find that current models struggle to follow the supplied semantics. Accuracy is highest on short Human-Written programs under standard semantics, but drops sharply under semantic shifts and on longer, structurally complex LLM-Translated and Fuzzer-Generated programs. Our results show that LLMs do not yet reliably reason from supplied formal semantics and still heavily rely on patterns learned in their pre-training.

We thank Cheng Ding, Ivan Grigorik, Yan Levin, Tong-Nong Lin, Karl Palmskog, Samuel Yuan, Linghan Zhong, and the anonymous reviewers for helpful feedback and discussions. Computational resources were provided by the Texas Advanced Computing Center (TACC²² 2 https://www.tacc.utexas.edu/) at the University of Texas at Austin, and AMD (University Program AI & HPC Cluster). This work was supported in part by the U.S. National Science Foundation (NSF) Nos. CCF-2217696, CCF-2313027, CCF-2403036; the NSF–Simons AI Institute for Cosmic Origins (CosmicAI³³ 3 https://www.cosmicai.org/) funded by NSF award AST-2421782; the Simons Foundation (MPS-AI-00010515); and a sponsored research award from Cisco. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsoring entities.