FIRST: Stale Rollouts Can Pick the Wrong Training Data
A Measurement-First Audit of Off-Policy Data Selection in RLVR

Anonymous

Introduction

RLVR pipelines choose their training data by scoring each candidate prompt with “how much would training on it help the current policy \(\pi\)?” and keeping the top \(k\). Fresh rollouts from \(\pi\) provide direct current-policy samples and avoid off-policy correction bias. They still contain Monte Carlo noise, however, and estimating every candidate requires enough responses per prompt after each policy update; this increases accelerator time and wall-clock latency and limits candidate-pool size. Reusing rollouts and rewards from an earlier behavior policy \(\beta\) reduces generation cost and supports broad candidate coverage and repeated ranking from stored artifacts, but introduces distribution shift and requires importance correction. The correction has two axes: (a) how the current policy reaches a token (prefix occupancy), and (b) how it finishes afterwards (continuation outcome). Several practical estimators deliberately correct at most one axis because correcting both multiplies hundreds of token ratios and can produce prohibitively large variance. The implicit assumption is that bias from the uncorrected component is too small to alter the ranking.

Published evidence already challenges this assumption. The method that popularized single-token influence scoring reports, in its own appendix, two relevant numbers: on- and off-policy per-prompt gradients agree pointwise (cosine above \(0.6\) on 40 of 50 prompts), yet only \(28.8\%\) of their top-\(10\%\) neighborhood structure is preserved [1]. We treat this as an observed discrepancy, not as proof of our mechanism. The result is not decomposed over the two correction axes, and other explanations—validation noise, difficulty clustering, score calibration—can also alter rank structure. This paper contributes a mechanism class that can produce exactly this dissociation, a decomposition that makes the explanations testable against each other, and a protocol for identifying the conditions under which the failure appears.

We examine this assumption at the estimator, selection, and evaluation levels. Our analysis identifies structural limitations beyond those described by a bias–variance account. Our claims come in three tiers:

1. Impossibility (proved). Either one-sided information set is compatible with opposite true top-\(k\) decisions, forcing minimax error at least \(1/2\) even at arbitrarily small policy divergence (Section 3). Group normalization does not help, and KL, ESS, and stale self-cosine remain benign. Cell disagreement is a useful unreliability signal, not a certificate.

2. Regime map (exploratory; rerun pending). Whether ranking reversal occurs depends on the task–capability regime (Section 5). A historical snapshot suggests three regimes: one where one-sided selection measurably loses precision, one where stale selection tracks the oracle, and one where saturation makes selection uninformative.

3. Measurement limits (proved + measured). Exact top-\(k\) identification obeys a gap-dependent \(\Omega(\sigma^2\Delta^{-2} \log(1/\delta))\) lower bound; a historical plug-in analysis placed its boundary radii far beyond the measured budget (Section 6). Independently, a decision-theoretic measurement ceiling caps what any oracle-noise-oblivious selector can demonstrate against a noisy oracle (Section 4).

Figure 1 summarizes the framework and how the sections fit together.

The central object of this paper is not an estimator but a measurement: the split-half oracle floor. The oracle itself is built from rollouts, so its own ranking carries noise. The floor quantifies that noise: split the oracle’s rollouts into two independent halves, rank the prompts with each half, and record the top-\(k\) agreement between the two rankings. A low floor means the pool offers little recoverable ranking signal at that rollout budget, and our ceiling result (Proposition 2) shows that no method conditionally independent of the held-out oracle noise can demonstrate more expected precision than the floor permits under the stated model. We applied this standard to our own pools by running FIRST (Floor-Informed Reliability of SelecTion), the pre-registered diagnosis of Section 8, and obtained a model-based budget-limited absence verdict: on saturated or tie-degenerate pools, a Spearman–Brown extrapolation up to \(K{=}256\) rollouts per prompt did not predict a stable signal regime. This historical verdict motivates the corrected measurement protocol; it is not treated as a current empirical finding until the reruns reproduce it.

One further observation applies to every experimental section. Selection evaluation tends to fail in the optimistic direction without producing any visible error. During this project our own pipeline generated four such evaluation artifacts, each of which initially looked like a publishable positive result, and later detected all four (Section 7). We document each mechanism together with its validity check, because the same mechanisms can arise in any experiment that reuses rollouts for data selection.

Related work

Off-policy influence and data selection.

Influence scores built on a single token’s importance ratio rank prompts from offline trajectories [1]. In our decomposition this is \(g_{00}\): neither axis is restored. The appendix of that work already reports the symptom that motivates our estimand: per-prompt gradients agree pointwise (cosine above \(0.6\) on 40 of 50 prompts), yet only \(28.8\%\) of the top-10% neighborhood structure is preserved. Corrections that reweight the visited prefix [2, 3] restore the occupancy axis (\(g_{10}\)). Their guarantees, however, assume a target-policy advantage, and practical critic-free objectives replace that advantage with a behavior-generated outcome reward; this substitution is the gap our counterexamples use. Corrections that reweight the next \(N-1\) sampled tokens [4] interpolate toward the outcome-corrected cell (\(g_{01}\)); they equal it only when \(N\) spans the remaining horizon. Full-trajectory corrections [5] restore both axes, but their variance grows with response length, so they serve as a cost reference rather than a practical selector. The prefix/continuation decomposition itself is therefore not new: prefix-normalized objectives separate the state ratio from the action ratio explicitly [35], and other two-factor splits decouple the correction across trust regions [36] or across the stages of asynchronous policy lag [37]. All of these target update stability; none treats the four cells as an estimator family for selection, and none asks whether the uncorrected axis can invert a prompt ranking. That inversion claim, not the decomposition, is what our counterexamples establish. The same tension appeared for Data Shapley in supervised data selection: valuation scores failed to outperform random selection until the selector was redesigned . A parallel negative result exists in preference learning: uncertainty-based active selection fails to beat random prompt selection in online DPO, while the proxy win rate and general capability drift apart [38]. That study observes the divergence but does not measure the reliability of the evaluation signal and gives no rule for deciding when selection is worth running. Our protocol (§8.1) is the RLVR counterpart of these audits, with one difference: the repair comes from a reliability measurement, not from a new estimator.

Rollout allocation and learning stability.

Recent selection methods combine influence, diversity, difficulty, and explorability [9]; compare subjective and objective uncertainty for query selection ; couple verifier outcomes to interpretable coverage [11]; use a single rollout’s hidden-state dynamics before training [12]; or score difficulty jointly with epistemic information . Prompt Replay instead reuses medium-difficulty prompts while regenerating on-policy trajectories [14]. Other methods allocate rollouts using predicted or observed reward variance [15, 16]. Gradient-alignment methods [6, 8] likewise optimize training yield. These works evaluate downstream learning efficiency, whereas our estimand is the reliability of the held-out top-\(k\) selection decision itself. Reported staleness tolerance does not contradict this estimand: evidence that GRPO training remains stable under large policy lag  concerns the loss and final performance, whereas our results concern the ranking of prompts. A robust loss does not imply a reliable ranking, and the measurement ceiling of Section 4 quantifies the distance between the two quantities. The practice we audit is current: recent selectors reuse past-rollout accuracies as priors with no staleness analysis , and difficulty schedulers state that accuracy estimates go stale across epochs yet treat the staleness as scheduling noise [42]. Selective importance sampling is another complementary response to stale data: it reduces token/trajectory correction mismatch rather than auditing prompt rankings . Classical off-policy correction and subset selection provide the statistical background: conditional importance sampling [18], stationary-distribution correction , doubly-robust policy gradients [20], and information-complexity lower bounds for top-\(k\) identification [21]. Our certification result (Section 6) is an empirical instance of the last of these: when value gaps at the ranking boundary vanish, identification cost diverges. A general budgeted algorithm for top-\(k\) certification also exists: it concentrates expensive queries at the ranking boundary and assumes the low-cost scores are valid up to noise [24]. Our results identify the case this assumption excludes. A stale one-sided score can be systematically biased (sign-flipped), and then certification fails through bias, not through budget. The two failure ingredients we study also appear in adjacent work on on-policy distillation [7]. There, degenerate repetition loops drive token-level agreement to near-zero divergence while outputs collapse, so a pointwise metric misses the object being consumed; and raw token log-ratios are dominated by the extreme tail, with the most negative 1% of tokens carrying roughly half the gradient. The archived v2 run’s reported \(0.1\%\) full-correction ESS is an analogous pipeline symptom, not the same statistic; because that run violated the generation contract, it is not a calibrated ESS estimate for our target correction.

Reliability of noisy rankings.

Top-\(k\) overlap under noisy scores has been analyzed directly in classical ranking work , which shows that top-\(k\) lists can be substantially less stable than global rank correlations. That work studies ranking noise itself, not stale-policy correction or a selector’s conditional agreement ceiling. Our floor and ceiling analysis imports split-half reliability and the Spearman–Brown prophecy [22, 23] into RLVR selection evaluation. Concurrent work applies the same split-half logic as a performance ceiling in a different domain (crowd reading) , and annotator-disagreement ceilings play the same role for subjective evaluation datasets [43]; our contribution is therefore the transplant of this reliability logic into RLVR selection evaluation and its pre-registered self-application, not the ceiling idea itself. The methods above do not report our specific split-half top-\(k\) oracle estimand, and Section 4 shows that several of our initially promising comparisons no longer hold once it is reported.

Theory: limitations of monitored metrics under one-sided correction

A \(2\times 2\) decomposition of stale-gradient estimators

For prompt \(x\), response \(Y=(A_1,\dots,A_T)\) with terminal verifiable reward \(R(Y)\), the current-policy gradient is \(g_{\pi}(x)=\sum_t \mathbb{E}_{H_t,A_t\sim\pi}[Q_t^{\pi}(H_t,A_t)\,z_t]\) with \(z_t=\nabla_\theta\log\pi(A_t\mid H_t)\). Throughout this subsection we assume a finite horizon \(T\), absolute continuity (\(\beta(Y)>0\) whenever \(\pi(Y)>0\)), exact unclipped likelihood ratios, and environment transitions that do not depend on the policy; implemented estimators that clip ratios for variance control are distinct from these population quantities. With behavior trajectories, token ratios \(r_j=\pi(A_j|H_j)/\beta(A_j|H_j)\), prefix products \(P_t=\prod_{j<t}r_j\) and suffix products \(S_t=\prod_{j>t}r_j\), four population quantities arise from the same stale data: \[\begin{aligned} g_{00}&=\textstyle\sum_t\mathbb{E}_\beta[r_t R z_t] &&\text{(occupancy }\beta\text{, continuation }\beta\text{)}\\ g_{10}&=\textstyle\sum_t\mathbb{E}_\beta[P_t r_t R z_t] &&\text{(occupancy }\pi\text{, continuation }\beta\text{)}\\ g_{01}&=\textstyle\sum_t\mathbb{E}_\beta[r_t S_t R z_t] &&\text{(occupancy }\beta\text{, continuation }\pi\text{)}\\ g_{11}&=\textstyle\sum_t\mathbb{E}_\beta[P_t r_t S_t R z_t]=g_{\pi}. && \end{aligned}\] The subscripts state which axis is restored. The exact error decompositions \[\begin{aligned} g_{10}-g_{\pi} &=\sum_t\mathbb{E}_{d_\pi,\pi}\!\big[(Q_t^{\beta}-Q_t^{\pi})z_t\big] &&\text{(continuation bias)},\\ g_{01}-g_{\pi} &=\sum_t\big(\mathbb{E}_{d_\beta,\pi}-\mathbb{E}_{d_\pi,\pi}\big) \!\big[Q_t^{\pi}z_t\big] &&\text{(occupancy bias)} \end{aligned}\] show that the two one-sided estimators remove different biases; they are not substitutes.

For a unit direction \(u\), write \(s_c(x)=u^\top g_c(x)\), \(c\in\{10,01\}\), and let \(\mathcal I_c^{(N)}\) denote everything retained by that one-sided score from \(N\) stale trajectories: its nonzero directional summands, their weights, and arbitrary summaries such as KL, ESS, norms, and self-similarity. A \(c\)-only selector is any possibly randomized top-\(k\) rule measurable with respect to \(\mathcal I_c^{(N)}\); it does not use the omitted ratio axis or fresh target rollouts.

Sign reversal at arbitrarily small divergence

The canonical two-token witnesses make the reversal explicit: for \(0<\epsilon<1/2\), each one-sided cell \(c\) has a construction with \[u^\top g_{\pi}=-\frac\epsilon2,\qquad u^\top g_c=+\frac\epsilon2,\qquad D_{\mathrm{KL}}(\pi\Vert\beta) =2\epsilon\log\frac{1/2+\epsilon}{1/2-\epsilon} =8\epsilon^2+O(\epsilon^4).\] Appendix 10 gives both pools, the general top-\(k\) embedding, and the group-normalization proof.

Empirically—an observation, not a theorem—in a seeded random search over \(5\times 10^{4}\) two-token, two-parameter environments we found zero instances of a double reversal (both one-sided estimators reversed against \(g_{\pi}\)) accompanied by cell agreement (\(\cos(g_{10},g_{01})>0.9\)); the identity \(g_{11}\equiv g_{\pi}\) was verified on every sample. This is a statement about the sampled proposal distribution, not a nonexistence claim: zero hits in \(5\times10^4\) draws bounds the event rate under that distribution below \(6\times10^{-5}\) (rule of three, \(95\%\)), and environments outside the sampled family are not covered.

Absence of disagreement is not a safety guarantee: Theorem 1 already rules out every guarantee based on one one-sided information set, and general parameter sharing permits both bias vectors to have adverse projections without agreeing with each other. We therefore use cell disagreement only as an unreliability signal that warrants further evaluation, never as a selector or certificate. The random search is proposal-dependent empirical evidence for the diagnostic value of disagreement, not a sufficiency theorem.

Measuring selection: floor, ceiling, and when comparison is meaningful

Estimand.

All experiments target the decision practitioners consume: the top-\(k\) set by \(s_i=\cos(\mu_i,v)\), where \(\mu_i\) is the prompt’s mean current-policy gradient and \(v\) a validation mean gradient. We report top-\(k\) precision against an on-policy oracle, boundary margins, and rollout budgets. Pointwise cosine averages are diagnostics, never primary results. In the corrected confirmatory protocol, stale rankers use the odd-indexed validation-gradient half and the evaluation oracle uses the disjoint even-indexed half; no validation-prompt gradient contributes to both sides of the primary ranking comparison.

The split-half floor.

The oracle itself is a \(K\)-rollout Monte Carlo estimate. We therefore report, for every run, the corrected split-half floor: candidate micro-groups and validation-prompt gradients are each split into disjoint odd/even halves, and each paired half induces a top-\(k\) set (ties broken by independent per-half jitter; Section 7 explains why this correction matters), and the floor is the mean overlap fraction over 20 jitter pairs. Chance is \(k/n\). Two scope notes. First, this is reliability under one fixed odd/even split of both candidate and validation samples; a confidence interval over prompts and rollout draws requires additional resampling. Second, the floor is not a lower bound on estimator performance—an estimator can measurably exceed it, so retention ratios (overlap divided by floor) can exceed \(1\) and are reported unclipped; their reading as “preserved fraction of available signal” holds only under the noise model of Proposition 2. Section 7 shows the floor is itself vulnerable to tie artifacts unless computed exactly this way.

A measurement ceiling.

The floor does more than describe the oracle: it bounds what any comparison against that oracle can show.

Table 1 gives the mapping at our experimental geometry (\(n{=}512\), \(k{=}51\)), validated by simulation (Appendix 11). Two consequences organize everything that follows. First, at \(\rho_h=0\) the ceiling equals chance \(k/n\) exactly: every oracle-noise-oblivious selector has chance expected precision, so oracle-overlap evaluation contains no ranking information. By continuity, a near-chance population floor permits only a small excess over chance. Second, a selector at the ceiling has exhausted all expected overlap available to any method satisfying the same held-out noise condition; exceeding it requires either more reliable oracle rollouts or information statistically coupled to the held-out oracle noise.

Retention and its domain.

When comparing regimes we report retention \((\mathrm{prec}-c)/(\mathrm{floor}-c)\) with \(c\) the chance level, declared undefined whenever \(\mathrm{floor}-c\le 0.02\); saturated cells otherwise produce sign-flipping or exploding ratios (the 14B GSM8K cell below is an example).

Historical regime map and corrected-rerun targets

Setup.

Qwen2.5-7B/14B-Instruct [28]; GSM8K [29], MATH-500 [30], and DAPO-Math-17k [31]; drift between \(\beta\) and \(\pi\) is controlled by LoRA  RFT for 50–400 steps; gradients use a fixed signed CountSketch projection ; the primary historical v2 oracle uses \(K{=}32\) fresh rollouts per prompt (8 micro-groups \(\times\) 4), while legacy scale confirmations retain the \(K\) recorded by their artifact manifest rather than being relabelled as \(32\); pools are \(n{=}256\) (v1) or \(n{=}512\) (v2) with \(k{=}10\%\). The v2 analysis criteria were pre-registered (equal per-cell budgets, seeded tie-breaks, and independent jitter), but a later audit found that the generation implementation did not satisfy the intended contract. The numerical tables below are therefore discovery snapshots from two seeds per task, not evidence for the stated target estimand. They are retained only to define analyses that corrected reruns must regenerate. The corrected protocol uses the raw model softmax (temperature \(1\), top-\(p{=}1\), top-\(k{=}0\), no repetition penalty) for both generation and likelihood-ratio recomputation; the archived runs instead inherited generation defaults that broke this identity. Empirical labels \(g_{00},g_{10},g_{01},g_{11}\) denote clipped variants: each complete token-level ratio product is clipped to \([0.1,10]\). Every prompt gradient uses the unstandardized leave-one-out advantage \(A_j=R_j-(K-1)^{-1}\sum_{\ell\ne j}R_\ell\). It is unbiased for the corresponding unclipped population gradient because the weighted score has zero expectation; after ratio clipping, the residual baseline term is part of the clipping bias. Lemma 1 addresses a different, standardized group-normalization convention [33] and is used only to show that normalization does not rescue the counterexample.

Discovery observations in nominal signal regimes

In the historical pre-contract v1 gate (7B GSM8K), one-sided estimators appeared to lose top-10% precision against the uncorrected baseline. At the largest drift (400 LoRA steps), the prefix-corrected estimator falls to precision \(0.040\) (overlap \(1/25\)). The exact hypergeometric tail for this cell is \(P(X\le 1)=0.27\), so we report it as a directional observation, not as an individually significant below-chance claim. One transparency note applies. The v1 floors were originally computed with the shared-jitter estimator that Section 7 later identified as failure mode P2, which had inflated them to \(0.32\). The tie-corrected re-estimation (Table 5) puts the drift-400 floor at \(0.206\). This is nominally a \(2\times\)-chance signal regime, and the \(0.040\) cell lies far below both floor and chance. Downstream, fine-tuned accuracy after selection (base \(0.740\)) spans \(0.820\)–\(0.880\) across oracle-, one-sided-, and random-selected pools, with no consistent ordering. In this saturation-adjacent regime, misranking has little room to change downstream accuracy, which is consistent with the results below.

On the archived 7B MATH-500 run the ordering flips. It has the largest historical floor (\(0.288\)), and its \(K'\)-curve rises (Table 5). Here the uncorrected \(g_{00}\) is worst (\(0.160\)), and every stale estimator retains only \(0.28\)–\(0.64\) of the available signal. At 14B MATH-500 the loss concentrates on the suffix axis (signal retention \(g_{01}{=}0.44\) vs. \(g_{10}{=}1.00\)), although the declining \(K'\)-curve there makes the absolute floor less certain. The discovery set therefore contains candidate regimes in which a cell failure and a benign stale-only dashboard co-occur. Because all these rows predate the generation-contract correction, they do not establish that this occurs under the paper’s target estimand. The corrected rerun tests precisely that event; Theorem 1 itself does not depend on the archived rows.

Where signal is weak, the oracle limits resolvable gains

The historical v2 snapshot put every completed condition at floors \(0.137\)–\(0.176\) against chance \(0.100\) (Table 2). Here stale overlap is above the random-selection null, although that test does not establish equivalence to the latent ranking. On GSM8K, the uncorrected stale baseline reaches precision \(0.294\)/\(0.235\) (seeds 1/2). Conditional on the archived rankings, upper-tail random-selection tails are \(2.8\times10^{-5}\) and \(2.1\times10^{-3}\) (their Fisher combination is \(1.0\times10^{-6}\)). These calculations describe separation from random within the archived artifact; because generation violated the estimand, they are not confirmatory \(p\)-values for the protocol claimed here. The value \(0.294\) is close to the plug-in model ceiling (\({\approx}0.30\) at floor \(0.137\); Table 1); a saturation claim requires propagating the finite-floor interval. At the point estimate, the stale estimator has exhausted nearly all overlap the model permits, so neither the losses of its one-sided competitors nor the gains of any other method are measurable in this regime. The DAPO pool is tie-degenerate: boundary margins are exactly \(0.0\), and precision itself moves by \(\pm 0.08\) under tie-break jitter, so we report bands. There, all stale estimators are at chance, and so is everything else.

Prompt-level reversal prevalence (preliminary).

Top-\(k\) overlap is a coarse summary. A reviewer will ask a finer question: how often does the sign of an individual prompt actually flip? Table 3 answers this for the completed v2 GSM8K seeds. Stale scores disagree in sign with the fresh oracle on \(36\)–\(48\%\) of signal-bearing prompts overall, on \(24\)–\(35\%\) of prompts inside the decision band, and on \(6\)–\(15\) of the oracle’s top-\(51\) prompts. These raw rates cannot be read against zero, because the oracle also disagrees with itself: an independent half of the oracle flips sign against the other half on \(39\%\)/\(43\%\) of the same prompts (decision band: \(27\%\)/\(20\%\)), as the reference row reports. Overall reversal for every cell lies within a few points of this reference. We therefore do not read the overall rates as estimator failure; they are dominated by oracle noise.

Two archived v2 contrasts initially looked stronger: one McNemar comparison gave \(p=0.014\), and disagreement-conditioned Fisher tests across the archived v2 runs gave \(p=0.0069\)–\(0.034\). They did not replicate in the later archived v3 MATH-500 snapshot: every McNemar comparison was non-significant (minimum \(p=0.23\) in seed 1 and \(0.52\) in seed 2), and the corresponding Fisher values were \(0.37\)/\(0.38\). Estimator reversal rates there spanned \(41\)–\(51\%\), overlapping the oracle self-disagreement anchors of \(44\)–\(48\%\). Both v2 and v3 predate the generation-contract fix and use different task/seed compositions, so this change cannot be attributed to the fix. It does, however, invalidate a replication or empirical-support claim. We therefore withdraw the v2 significance claims and retain disagreement only as a pre-registered fresh-audit trigger motivated by Proposition 1. For scalar per-prompt scores, a double reversal also appears as cell agreement, so agreement remains explicitly non-certifying. DAPO contributes only \(2\)–\(4\) signal-bearing prompts per seed and is excluded from these contrasts.

Where capability saturates, selection degenerates

At 14B on GSM8K (\({\sim}95\%\) accuracy), the pool contains \(132/256\) prompts with zero reward variance. Every estimator’s precision equals the chance level (\(0.087\)–\(0.111\) vs. \(0.098\)), and the oracle disagrees with itself below chance (floor \(0.080<0.098\)). Selection, ranking, and certification therefore become uninformative in this snapshot. The practical consequence is that selection pipelines must scale candidate difficulty with model capability; otherwise they spend, without noticing, on machinery that cannot change the outcome.

A gap lower bound for exact certification

Even where a point estimate selects well, one may ask for the decision to be certified: spend fresh rollouts near the ranking boundary until confidence intervals separate the top-\(k\) set. We implemented a boundary refinement diagnostic with candidate/evaluation sample separation (Appendix 12). It allocates \(\delta\) over all possible adaptive looks, but its default isotropic-Gaussian radius remains model-based; we do not claim distribution-free coverage. The inverse-square lower bound below is independent of that diagnostic.

The bound already holds in the two-boundary-prompt Gaussian subclass. The change-of-measure proof is in Appendix 12; adaptive allocation can remove off-boundary waste but not the \(\Delta^{-2}\log(1/\delta)\) dependence.

The historical diagnostic returned a corresponding exploratory negative. Across its completed conditions (both scales, three tasks, drift 50–400, selection fractions 5–25%), the stopping criterion was not met at budgets \(1.02\)–\(1.04\times\) the uncorrected cost. The reason is geometric: top-\(k\) boundary margins are \(0.00\)–\(0.24^\circ\), while achievable confidence radii are \(51\)–\(180^\circ\). The gap is not a constant factor. With radius \(\alpha_v(m)=\arcsin(c/\sqrt m)\) under the instrument’s iid Gaussian approximation, reaching the observed margins requires \(4.1{\times}10^{5}\) to \(1.0{\times}10^{8}\) times more observations—5–8 orders of magnitude in budget. These numbers are instrument-specific plug-in factors; Proposition 3 establishes that their inverse-square dependence is algorithm-independent in the Gaussian subclass. They motivate the theorem but do not establish finite-sample coverage or a universal impossibility result. Approximate-membership, indifference-zone, and time-uniform formulations are weaker demands this snapshot does not exclude.

Four evaluation failure modes identified in our pipeline

Every mechanism below produced, at some point in this project, an apparently valid positive result that survived a first review and failed under a targeted audit. We report them as findings in their own right. Each comes with a validity check now built into our protocol, and we suspect instances of each exist in the current rollout-reuse literature.

P1: Oracle-sample reuse (leakage).

Our v1 hybrid intervention (completing the “missing half” with fresh continuations) showed a perfect 21/21-condition recovery, mean \(+0.46\)/\(+0.49\) precision—which would have been the central causal result of the paper. A code audit found the treated cell reused rollouts that also defined the oracle; the shared noise, not causation, produced the recovery. A later disjoint-sample replay removed the apparent recovery. A subsequent audit also found unequal continuation horizons in the mixed cells, so neither legacy hybrid result supports a causal claim. Check: strict sample disjointness, equal rollout count, a common total response horizon, and a pre-registered decision cut of \(0.5\) (other cuts are sensitivity analyses), all enforced by the current code.

P2: Shared-jitter tie inflation.

Our first floor implementation broke score ties with a jitter shared across the two halves. On a tie-degenerate pool (DAPO, margins \(0.0\)) this inflated the floor from \(0.176\) to \(0.804\), which briefly supported an incorrect “inverse correlation” conclusion. The same estimator had also inflated the v1 floors (\(0.32\) reported vs. \(0.15\)–\(0.21\) corrected), which means every absolute floor in this literature computed with tie-unaware agreement warrants re-examination. Check: independent per-half jitter streams, 20 seeded pairs, plus reporting precision bands under jitter on tie-heavy pools.

P3: Cell-budget imbalance.

Comparing estimator cells with unequal rollout counts or response horizons favors the cell with the larger budget without any visible sign of it; our v1 hybrid grid had this flaw in addition to P1. Check: equal-\(K\), equal-horizon redesign of all cell comparisons, verified from manifests.

P4: Saturation-shared overlap inflation.

An above-chance floor can be an artifact of shared liveness rather than shared ranking: prompts that saturate in both halves (all-correct or all-wrong) drop to an exactly-tied score, and the surviving “live” sets overlap between halves, producing top-\(k\) agreement with zero true ranking signal. In a synthetic pool with no ranking signal, saturation sharing alone yields floor \(0.126\) at \(K'{=}8\), decaying toward chance (\(0.110\)) at \(K'{=}32\). The signature of this artifact is a floor that decreases as the oracle budget grows. The archived GSM8K \(K\)-scaling curves (Section 8) have this decreasing shape. Because those runs are pre-contract, this is a diagnostic signature to retest, not a current estimate of the population floor. Check: always report the floor as a function of \(K'\); treat a non-increasing floor curve as an artifact indicator, and report tie mass alongside.

FIRST: diagnosing signal absence from existing artifacts

These failure modes motivate the final contribution: FIRST, a pre-registered audit of whether a pool supports selection. It is artifact-only after an instrumented run has stored independent oracle micro-groups; otherwise those measurements require a fresh probe. It has three stages.

Stage 1: conditional-subset precheck.

Any proposed pool intervention definable as a filter on existing prompts (e.g. “keep only live prompts, \(0<\text{pass-rate}<1\)”) already exists as a subset of completed runs. Computing the corrected conditional floor on that subset (with bootstrap CIs) prechecks the intervention at zero GPU cost: if the conditional floor does not clear \(2\times\) chance in a majority of eligible runs, building the new pool has no evidential basis. On the archived pools this returned NO-GO, and the intervention (a live-only GSM8K pool) was cancelled without spending a GPU-hour. The corrected runs must recompute that verdict.

Stage 2: \(K\)-scaling floor curve.

From stored micro-group gradients the floor is recomputed over seeded disjoint micro-group partitions at \(K'{=}8,16,24,32\), and extrapolated to \(K'{=}64\)–\(256\) by Spearman–Brown on the half-reliability \(r_1\) mapped through the Gaussian overlap function, with a calibration check at the largest observed \(K'\). The pre-registered verdict rule: if a majority of eligible runs are predicted to reach \(2\times\) chance by \(K'\le 256\), recommend oracle expansion; otherwise report budget-limited absence under the extrapolation model.

Table 4 shows the historical result: one GSM8K seed extrapolates to \(2\times\) chance at \(K'{=}128\) from \(r_1{=}0.045\); the other is flat at chance; both DAPO runs never reach it. The historical majority was not met, so the procedure returned budget-limited absence within those archived pools and the extrapolated budget range. This is not a current verdict: the generation-contract-corrected reruns must recompute the vote.

Running the same historical analysis over every then-complete run in the project (twelve conditions including the v1 gate family; Table 5) widens the discovery set without changing the historical vote. Nominal signal regimes appeared in 2–3 of 12 conditions. The two strongest rows coincided with candidate one-sided losses (7B MATH-500 and maximum-drift GSM8K), while half of the observed curves declined in \(K'\). These are rerun targets, not population estimates.

Stage 3 (replay): budget-space stress test.

As a historical exploratory check we replayed fresh-budget allocation policies against a sample-disjoint truth (odd micro-groups only): stale cells, a pass-rate heuristic, random, uniform and boundary audits at 1–25% budgets, a sequential disagreement-guided audit, full-fresh at up to the available selection-half budget, and our own floor-gated rule. In the v2 snapshot, no fresh budget improved weak-signal GSM8K: full-fresh at \(100\%\) scored \(0.167\) against \(0.235\) for stale and \(0.255\) for behavior pass rate; floor-gated scored \(0.172\). The later v3 MATH-500 snapshot gave the same policy-level pattern with different values: stale \(0.275\), pass rate \(0.265\), full-fresh \(0.225\), and floor-gated \(0.212\). Neither snapshot validates a policy because both predate the generation-contract fix, and the seed-level cell orderings and confidence intervals were unstable. They do rule out presenting the floor-gated rule as already validated; FIRST is retained as an audit whose branches below are hypotheses for corrected evaluation. The archived DAPO rows are tie-degenerate (truth margins \(0.0\)). The v2 full correction also reported effective sample size \(0.1\%\) and clip fraction \(0.94\)–\(1.00\), but the generation/likelihood identity did not hold, so these quantities cannot be interpreted as estimates of the target correction’s variance.

Two caveats accompany this historical analysis. First, the calibration check shows that the Gaussian model under-predicts observed floors at \(K'{=}32\) (e.g. \(0.094\) predicted vs. \(0.156\) observed). The mismatch is compatible with P4 but does not identify its cause, and it prevents treating the extrapolated vote as calibrated. Second, the single positive extrapolation rests on an \(r_1\) small enough to be affected by the P4 artifact, so we treat it as motivation for a corrected re-check, not as grounds for a GPU expansion.

From diagnosis to testable decision hypotheses

The diagnostic verdict defines testable policy hypotheses, not validated deployment rules. The historical replay estimates their costs, while a corrected, independently evaluated rerun must determine whether any branch improves downstream learning. The data-valuation literature encountered the same problem in supervised learning: when Shapley-based selection proved no better than random, the response was a corrected selection method, not abandonment of selection [26, 27].

Practical guidance, limitations, and outlook

For practitioners.

(1) Measure the floor first—run the FIRST audit and pre-register the comparison suggested by its branch (§8.1). Treat \(2\times\) chance as a heuristic trigger, not a confidence threshold, and report probe cost explicitly. (2) Compute the boundary lower bound before budgeting certification; the historical 5–8 order-of-magnitude plug-in gap motivates caution, and Proposition 3 shows that adaptive allocation cannot remove its inverse-square dependence. (3) Scale candidate difficulty with capability, or selection degenerates regardless of estimator. (4) Report floor and ceiling with any selection comparison; a claimed improvement above the measurement ceiling of its own oracle indicates assumption violation, estimator–oracle dependence, or an evaluation artifact. (5) Distrust one-sided rankings when the two one-sided cells disagree; agreement is not sufficient for reliability, while disagreement provides an unreliability signal without additional rollouts, formalized as branch (c) of the protocol.

Limitations.

The quantitative snapshot covers Qwen2.5 (7B/14B), math tasks, and LoRA drift, with two v2 seeds, but predates the corrected generation and validation-split contracts. It is exploratory until the current code regenerates the tables. The ceiling proposition assumes iid latent Gaussian scores, equal-variance parallel oracle noise, and conditional independence from held-out oracle noise; simulation validates the resulting map at our geometry, and a distribution-free ceiling remains open. The disagreement result is an existence-plus-search finding, not a general sufficiency theorem. The theory counterexamples are two-token constructions by design: they show what the monitored metrics cannot exclude, not what typically happens, and the regime map is their empirical complement.

Safety relevance.

Data selection is a curation channel: whatever a misranked estimator systematically promotes is amplified by subsequent training. Corollary 1 therefore has a safety reading. If stale selection promotes prompts on which shortcut or reward-hacking behavior scores well, the drift metrics a team monitors cannot flag the promotion, for the same reason they cannot flag the sign reversal itself. The selection layer is then an unaudited input to the reward loop. Theorem 1 transfers to a safety target whenever the same trajectory, support, and verifiable-reward assumptions hold. A safety application must separately audit partial observability and adversarial or misspecified rewards before using the measurement protocol.

Outlook.

The negative results are central rather than incidental. They state the conditions under which stale-rollout data selection can work—unsaturated, tie-free, signal-bearing pools, which our diagnosis tests from stored independent artifacts or a fresh probe—and the conditions under which it cannot. The measurement-first protocol (floor, ceiling, preregistration, and the four validity checks) applies to any selection-method paper, including ours.

Counterexample constructions and machine verification

Continuation construction.

This reverses \(g_{00}\) and \(g_{10}\) in a two-token MDP. The first action \(A_1\sim\mathrm{Bernoulli}(q)\), \(q{=}1/2\), identical under both policies; reward \(R=A_2\). Behavior success profile \(Q^\beta(A_1{=}0)=\frac12-\epsilon\), \(Q^\beta(A_1{=}1)=\frac12+\epsilon\); current policy swaps the two values. The gradient w.r.t. the logit of \(q\) is \(g_{\pi}=-\epsilon/2\) while \(g_{00}=g_{10}=+\epsilon/2\): the prefix already matches perfectly, so a perfect prefix ratio changes nothing, and the behavior-policy ending drives the sign. Mean suffix KL is \(O(\epsilon^2)\); the 1-D cosine is \(-1\) throughout.

Occupancy counterexample (reverses \(g_{00},g_{01}\)).

Behavior visits \(P(A_1{=}0)=\frac12+\epsilon\), current \(\frac12-\epsilon\); the second action is \(\mathrm{Bernoulli}(1/2)\) under both, and \(R=A_2\) if \(A_1{=}0\), else \(1-A_2\). For the second-action logit, \(g_{\pi}=-\epsilon/2\) while \(g_{00}=g_{01}=+\epsilon/2\): no tokens remain after \(A_2\), so the future ratio is exactly \(1\), and the stale visit distribution drives the sign. Trajectory KL is again \(O(\epsilon^2)\).

In both cases the changed Bernoulli has \[D_{\mathrm{KL}}(\pi\Vert\beta) =2\epsilon\log\frac{1/2+\epsilon}{1/2-\epsilon},\] which is finite exactly for \(0<\epsilon<1/2\) and has expansion \(8\epsilon^2+O(\epsilon^4)\).

Indistinguishable pools.

We prove the \(c=10\) case; \(c=01\) is its time reversal. Add an exogenous initial branch \(B\in\{\mathrm{visible},\mathrm{hidden}\}\) with hidden probability \(h\). In both branches \(A_1\sim\mathrm{Bernoulli}(1/2)\) and the direction \(u\) controls only its logit. On the visible branch of prompt \(i\), behavior and target share \[P(A_2=1\mid A_1=1)=\tfrac12+\nu_i,\qquad P(A_2=1\mid A_1=0)=\tfrac12-\nu_i.\] On the hidden branch behavior uses \(P_\beta(A_2=1\mid A_1)=1/2\), while target world sign \(s_i\in\{-1,+1\}\) uses the same display with \(\nu_i\) replaced by \(s_i\epsilon\). Reward is \(R=A_2\). The retained prefix/current weight for the directional \(g_{10}\) summand is identically one, and \[u^\top g_{10}(x_i)=\frac{(1-h)\nu_i}{2},\qquad u^\top g_{\pi}(x_i)=\frac{(1-h)\nu_i+h s_i\epsilon}{2}.\] Choose \(0<\nu_2<\nu_1\) with \((1-h)\nu_1<h\epsilon\). World \(W_0\) takes \((s_1,s_2)=(+1,-1)\) and \(W_1\) takes \((-1,+1)\). The strict one-sided ranking is \(x_1>x_2\) in both worlds, whereas the true top-1 swaps; in \(W_1\) the selected prompt \(x_1\) also has negative true directional gradient.

The behavior law, retained weight, and every retained directional summand are identical across worlds. The hidden target continuation has the same KL for signs \(+1\) and \(-1\) by Bernoulli symmetry, so supplying exact policy KL does not distinguish them either. The KL is \(hO(\epsilon^2)\) and can be made below any \(\delta\). Therefore the full \(N\)-sample \(10\)-only information has the same law for every \(N\). If a randomized selector chooses \(x_1\) with probability \(p\), its errors in \(W_0,W_1\) are \(1-p\) and \(p\); their maximum is at least \(1/2\). The same ambiguity applies to arbitrary top \(k\) by adding \(k-1\) prompts that dominate in both worlds and \(n-k-1\) prompts that are dominated in both.

For \(c=01\), let the direction control only the second-action logit, use \(A_2\sim\mathrm{Bernoulli}(1/2)\) under both policies, and set \(R=A_2\) after \(A_1=0\) and \(R=1-A_2\) after \(A_1=1\). The visible branch shares \(P(A_1=0)=1/2+\nu_i\); the hidden branch has behavior probability \(1/2\) and target probability \(1/2+s_i\epsilon\). Then the retained current/suffix weight is one and the same two displayed gradients result. This proves the theorem for both axes.

Measurement ceiling: derivation sketch and validation

Fix the latent vector \(t\) and define \(p_i(t)=P(i\in\operatorname{TopK}(o)\mid t)\). If \(t_i\ge t_j\), exchangeability and equal noise variance imply \(p_i(t)\ge p_j(t)\). One direct coupling swaps the iid noises of \(i\) and \(j\): every realization in which only the lower-mean \(j\) enters the top \(k\) maps injectively to one in which only \(i\) enters. Therefore the size-\(k\) subset maximizing conditional expected overlap is \(\operatorname{TopK}(t)\).

Given \(t\), any \(S(Z)\) satisfying \(Z\perp(e_a,e_b)\mid t\) is merely a random mixture of size-\(k\) subsets. Hence \[\mathbb E[|S(Z)\cap\operatorname{TopK}(o)|\mid t] \le \sum_{i\in\operatorname{TopK}(t)}p_i(t).\] Averaging proves the first inequality in Proposition 2. The parallel-measurement model gives \(\rho_h=\tau^2/(\tau^2+\sigma^2)\) and \(\rho_f=\tau^2/(\tau^2+\sigma^2/2)=2\rho_h/(1+\rho_h)\). Thus \((t_i,o_i)\) is bivariate Gaussian with correlation \(\sqrt{\rho_f}\), while \((a_i,b_i)\) has correlation \(\rho_h\), which gives the ceiling and population floor through the same Gaussian overlap function. This overlap is monotone in nonnegative correlation: for each top-\(k\) membership event \(A_i\), its Gaussian noise stability has Hermite expansion \(P(X\in A_i,Y\in A_i)=\sum_{m\ge0}\rho^m\|f_{i,m}\|_2^2\), whose coefficients are nonnegative; summing over \(i\) gives \(\mathrm{Ovl}_{n,k}(\rho)\).

Simulation at \(n{=}512\), \(k{=}51\) (600–800 replicates per cell; independent-jitter tie handling) produces Table 1. The table is a population-model map, not an equality for one finite observed overlap. A future confirmatory analysis must propagate a valid floor interval through it. Heavy-tailed or heteroscedastic oracle noise can violate the model ceiling and must be tested separately. This distributional issue is distinct from the P4 saturation/tie artifact.

Boundary-refinement diagnostic

The implemented diagnostic does not use off-policy scores as a statistical prior: every candidate receives a small fresh micro-group; fresh-gradient scores carry model-based confidence balls with a union bound over every possible adaptive look; fresh rollouts are then spent only on candidates whose balls straddle the top-\(k\) boundary, until \(\min L_{\text{selected}}>\max U_{\text{rest}}\) or the budget is exhausted, in which case the stopping criterion fails. The look-wise union bound handles optional stopping, but the isotropic-Gaussian ball itself is an approximation; this implementation is therefore a cost diagnostic rather than a distribution-free certificate. The lower bound below concerns any genuinely \(\delta\)-correct algorithm and remains valid independently.

Proof of Proposition 3.

Keep every non-boundary mean fixed and form an alternative instance by swapping \(\mu_{(k)}\) and \(\mu_{(k+1)}\). Let \(A\) be the event that the algorithm includes the original \(k\)th prompt. Uniform \(\delta\)-correctness gives \(P(A)\ge1-\delta\) on the original instance and \(P'(A)\le\delta\) on the alternative. Data processing from the adaptive transcript to the final decision gives \[\mathrm{KL}(P_{\mathrm{tr}}\Vert P'_{\mathrm{tr}}) \ge \operatorname{kl}(1-\delta,\delta).\] Each observation from either boundary prompt changes its Gaussian mean by \(\Delta\) and contributes \(\Delta^2/(2\sigma^2)\) KL; observations from every other prompt contribute zero. The adaptive chain rule therefore gives \[\mathrm{KL}(P_{\mathrm{tr}}\Vert P'_{\mathrm{tr}}) =\frac{\Delta^2}{2\sigma^2} \mathbb E[N_{(k)}+N_{(k+1)}],\] which rearranges to the claimed lower bound.

Preregistration and audit trail

All v2 judgment criteria, the precheck GO/NO-GO rule, the \(K\)-scaling verdict rule, and the reframing decision tree (“structural absence \(\Rightarrow\) this paper’s structure”) were registered in the project log before the corresponding results were computed; the four failure modes each carry a before/after commit pair. The code release contains the verification scripts, artifact schemas, and audit record. Corrected raw artifacts and generated tables must be archived with the confirmatory version.

A pre-submission prior-art search (2026-08-20) queried arXiv along the three contribution axes: the one-sided importance decomposition and its variants, stale-signal prompt selection, and reliability ceilings for noisy evaluation. Candidates with potential overlap were read in full rather than judged from titles. We found no prior work that jointly states the four-cell estimator family, proves its ranking-inversion result, and audits stale RLVR prompt rankings against an independent split-half fresh oracle; nearby work on the individual components is cited and distinguished in Section 2 (; [38]; [39]; [40]; [44]). The full query log and per-paper verdicts are archived in the repository (PRIOR-ART-2026-08-20.md).

References

1. Erle Zhu et al. Data Efficient RLVR via Off-Policy Influence Guidance. ACL 2026. https://aclanthology.org/2026.acl-long.2141/

2. Shiye Lei, Zhihao Cheng, and Dacheng Tao. A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization. https://arxiv.org/abs/2601.22718

3. Yuheng Zhang et al. Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective. https://arxiv.org/abs/2605.07331

4. Deokgyu Yoon et al. Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards. https://arxiv.org/abs/2605.20865

5. Lei Pang, Jun Luo, and Ruinan Jin. TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback. https://arxiv.org/abs/2508.02833

6. Ningyuan Yang et al. GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning. https://arxiv.org/abs/2602.21492

7. Zichao Yu et al. Mismatch Matters: On-Policy Distillation Beyond Token Agreement. https://arxiv.org/abs/2608.09836

8. Shikun Li et al. LearnAlign: Reasoning Data Selection for Reinforcement Learning in Large Language Models Based on Improved Gradient Alignment. https://arxiv.org/abs/2506.11480

9. Xinyu Tang et al. Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward. ICLR 2026. https://openreview.net/forum?id=sruA4AZmZI

10. Hao Yi et al. Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR. ICLR 2026. https://openreview.net/forum?id=OOTokVgBY6

11. Yuhan Li, Mingxu Zhang, Dazhong Shen, and Ying Sun. IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage. https://arxiv.org/abs/2605.28247

12. Jianghao Wu et al. Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection. ICML 2026. https://arxiv.org/abs/2605.28631

13. Xinyu Zhou et al. Efficient RLVR Training via Weighted Mutual Information Data Selection. https://arxiv.org/abs/2603.01907

14. Andrei Baroian and Rutger Berger. Prompt Replay: Speeding Up GRPO with On-Policy Reuse of High-Signal Prompts. https://arxiv.org/abs/2603.21177

15. Hieu Trung Nguyen et al. Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards. https://arxiv.org/abs/2602.01601

16. Heyang Jiang, Henry Liu, and Baharan Mirzasoleiman. Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning. https://arxiv.org/abs/2607.22002

17. Yu Li et al. Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment. https://arxiv.org/abs/2607.04728

18. Mark Rowland et al. Conditional Importance Sampling for Off-Policy Learning. AISTATS 2020. https://proceedings.mlr.press/v108/rowland20b.html

19. Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-Policy Policy Gradient with Stationary Distribution Correction. UAI 2020. https://proceedings.mlr.press/v115/liu20a.html

20. Jiawei Huang and Nan Jiang. From Importance Sampling to Doubly Robust Policy Gradient. ICML 2020. https://proceedings.mlr.press/v119/huang20b.html

21. Emilie Kaufmann and Shivaram Kalyanakrishnan. Information Complexity in Bandit Subset Selection. COLT 2013. https://proceedings.mlr.press/v30/Kaufmann13.html

22. Charles Spearman. Correlation Calculated from Faulty Data. British Journal of Psychology, 1910.

23. William Brown. Some Experimental Results in the Correlation of Mental Abilities. British Journal of Psychology, 1910.

24. Lutz Oettershagen. Top-\(k\) on a Budget: Adaptive Ranking with Weak and Strong Oracles. https://arxiv.org/abs/2601.20989

25. Kazuki Nakayashiki and Keisuke Watanabe. Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict? https://arxiv.org/abs/2608.01704

26. Jiachen T. Wang et al. Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits. ICML 2024. https://proceedings.mlr.press/v235/wang24cg.html

27. Xiao Tian et al. Is Data Shapley Not Better than Random in Data Selection? Ask NASH. https://arxiv.org/abs/2605.10684

28. An Yang et al. Qwen2.5 Technical Report. https://arxiv.org/abs/2412.15115

29. Karl Cobbe et al. Training Verifiers to Solve Math Word Problems. https://arxiv.org/abs/2110.14168

30. Dan Hendrycks et al. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/abs/2103.03874

31. Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476

32. Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685

33. Zhihong Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300

34. Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding Frequent Items in Data Streams. ICALP 2002.

35. Wenhao Zhang et al. Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning. https://arxiv.org/abs/2608.01418

36. Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, and Xiangyang Ji. RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning. https://arxiv.org/abs/2606.01281

37. Zhong Guan et al. Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction. https://arxiv.org/abs/2605.12070

38. Giyeong Oh, Junghyun Lee, Jaehyun Park, Youngjae Yu, Wonho Bae, and Junhyug Noh. Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs. https://arxiv.org/abs/2604.02766

39. Minghao Tian, Yunfei Xie, and Chen Wei. How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning. https://arxiv.org/abs/2605.17570

40. Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? https://arxiv.org/abs/2510.01161

41. Shuang Liang, Haoyang Zhou, Yifan Gong, Guowei Wang, and Xiting Wang. LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models. https://arxiv.org/abs/2607.28077

42. Dohyung Kim, Minbeom Kim, Jeonghye Kim, Sangmook Lee, Sojeong Rhee, and Kyomin Jung. Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVR. https://arxiv.org/abs/2602.12642

43. Fredrik Cumlin. Rho-Perfect: Correlation Ceiling For Subjective Evaluation Datasets. https://arxiv.org/abs/2602.08552

44. Or Zuk, Liat Ein-Dor, and Eytan Domany. Ranking Under Uncertainty. UAI 2007, pp. 466–473. https://arxiv.org/abs/1206.5280