The Data Gradient: Quantifying Dataset Influence in End-to-End Machine Learning Systems

Khadija Kamene
Mount Kenya University, Mombasa, Kenya
Correspondence to: Khadija Kamene, hadijashah@yahoo.com

DOI: https://doi.org/10.70389/PJCS.100014

Additional information

• Ethical approval: N/a
• Consent: N/a
• Funding: No industry funding
• Conflicts of interest: N/a
• Author contribution: Khadija Kamene – Conceptualization, Writing – original draft, review and editing
• Guarantor: Khadija Kamene
• Provenance and peer-review: Unsolicited and externally peer-reviewed
• Data availability statement: N/a

Keywords: Data-centric ai pipeline, Gradient-based dataset influence, Hessian-vector approximation, Influence-guided dataset refinement, Per-sample data valuation.

Peer Review
Received: 12 March 2026
Last revised: 02 June 2026
Accepted: 09 June 2026
Version accepted: 6
Published: 16 June 2026

Plain Language Summary Infographic

Introduction

When a well-architected model underperforms, the instinctive response is to adjust the architecture, tune hyperparameters, or collect more data. Less commonly asked is which specific training samples are driving the model toward or away from good generalization. Mislabeled, redundant, or systematically biased samples can silently degrade performance in ways that additional model capacity cannot fix. Yet, despite growing recognition of data quality as a first-order concern, methods for rigorously attributing model behavior to individual training instances remain either computationally intractable at scale or disconnected from practical training workflows.

The most influential line of work on this problem traces back to classical statistical influence functions, formalized for modern neural networks by Koh and Liang,¹ who showed that the effect of removing a single training point could be approximated via the gradient of that sample and the inverse Hessian of the training loss. TracIn² extends this by summing gradient dot products across checkpoints. Representer point selection³ decomposes predictions through the representer theorem. Data maps⁴ characterize samples by training dynamics. Each method offers genuine insight, but each operates largely as a standalone diagnostic rather than an embedded component of an active refinement pipeline. Recent work on efficient influence estimation—including Hessian-free, low-rank, and sketching-based approaches^5,6—and analyses of the validity of second-order approximations in deep networks⁷ further motivate the need for practically deployable frameworks that integrate these advances.

The data gradient framework builds directly on Koh and Liang.¹ We make no claim to a fundamentally new influence estimator. Our contribution lies in three areas: (i) embedding influence estimation into an end-to-end pipeline supporting iterative sample removal, label correction, and reweighting without offline reprocessing; (ii) introducing the dataset contribution score (DCS), a normalized aggregate metric tracking dataset quality across refinement iterations; and (iii) providing empirical validation across image, text, and tabular data with both neural and tree-based models, with explicit treatment of the approximations required for non-differentiable architectures.

Influence estimation for tree-based models—­Random Forests and gradient boosted trees (GBTs)—presents a non-trivial challenge, as these models lack the smooth, differentiable loss surfaces assumed by classical influence functions. We address this by deriving a pseudo-residual-based approximation for GBTs with explicit local-linearity justification and leave-one-out (LOO) calibration, and a proximity-based heuristic for Random Forests with appropriately narrowed claims and failure-case analysis.

The remainder of this paper is organized as follows: Section “Literature Review” reviews related work; Section “Methodology” presents the mathematical formulation; Section “Pipeline Architecture and Pseudocode” describes the pipeline architecture and perturbation strategies; Section “Experimental Design” details the experimental design; Section “Results” reports the results; and Section “Discussion” discusses implications, limitations, and future directions. Appendix A provides a pipeline diagram and complete pseudocode; Appendix B provides full training and influence computation configurations; and Appendix C presents ablation study results.

Literature Review

Data-Centric AI

The dominant paradigm in machine learning has long been model-centric. Sambasivan et al.⁸ documented empirically that data quality issues were a primary source of failure in deployed AI systems; yet, data work remained undervalued. Northcutt et al.⁹ demonstrated that confident learning could identify mislabeled samples at scale, revealing non-trivial error rates in widely used benchmarks. Surveys of data-centric methods^10,11 have highlighted that dataset quality improvements often yield larger performance gains than equivalent investments in model complexity.

Influence Functions and Sample-Level Attribution

Koh and Liang¹ showed that the parameter change from upweighting sample i by ε is Δθ ≈ −ε Hθ −1 gi, composable with the validation gradient to yield a scalar influence score. Tractability relies on CG or LiSSA12-based inverse Hessian approximations. TracIn² sums gradient dot products across checkpoints—computationally lighter but sensitive to checkpoint frequency; we ablate checkpoint frequency for TracIn in Section “TracIn Checkpoint Frequency Ablation”. Representer point selection³ provides exact attribution for L2-regularized models via the representer theorem, but is primarily suited to neural network final layers.

Recent work has extended influence estimation in several directions. FastIF and related Hessian-free methods⁵ reduce the cost of the inverse-Hessian-vector solve through low-rank approximations and sketching, making full-model influence tractable for moderately large networks. Incremental influence methods¹³ update influence scores as new data arrive without full recomputation, enabling streaming applications. Several analyses⁷ have examined when the second-order approximation underlying classical influence functions remains reliable in overparameterized networks, finding that reliability degrades with non-convexity and that LOO calibration is essential for empirical validation—as we show in Section “Leave-One-Out Calibration.”

Dataset Dynamics

Data maps4 track the confidence and variability of model predictions across epochs, identifying easy, hard, and ambiguous samples without Hessian computation. Ambiguous samples were often the most informative for generalization—a finding complementary to influence-based pruning strategies explored here.

Dataset Valuation

Data Shapley14 defines sample value as the average marginal contribution across all dataset subsets, satisfying axiomatic properties but requiring exponentially many model evaluations. Practical approximations^15,16 improve scalability but remain substantially more expensive than influence function approaches. Our framework can be distinguished from data valuation approaches used within iterative pipelines: Shapley-based methods optimize an axiomatically^17,18 grounded global value function, whereas the data gradient framework optimizes a local, gradient-based influence signal with respect to a fixed validation objective, a distinction that makes the latter more computationally tractable for iterative refinement but less robust to distributional shift. For tree ensembles specifically, leaf-influence methods¹⁹ and Tree SHAP20 provide principled attribution directly from tree structure without gradient approximations; we perform a comparison against these baselines in Section “Baselines.”

Limitations of Existing Approaches and Positioning

Influence functions and TracIn^1,2 are designed for differentiable models. Shapley methods14 are prohibitively expensive for iterative refinement. Data maps4 do not quantify influence on a specific validation objective. Tree-specific methods^19,20 handle non-differentiable architectures but are not integrated into end-to-end pipeline refinement loops. The data gradient framework addresses these gaps by combining the influence function estimator with pipeline integration, an aggregate quality metric, and explicit if approximate treatment of tree-based models.

Methodology

Overview and Notation

Let D = {(xi, yi)}ki=₁ denote the training dataset, Dval the held-out validation set, and θ ∈ ℝR the model parameters minimizing empirical loss L(θ; D) = (1/N) Σi ℓ(θ; xi, yi), where ℓ denotes the per-sample loss function. The per-sample gradient is gi = ∇θ ℓ(θ; xi, yi) and the Hessian of the training loss is Hθ = (1/N) Σj ∇2θ ℓ(θ; xj, yj). The validation gradient is v = ∇θ L(θ; Dval). All quantities are evaluated at the converged parameter estimate θˆ obtained after training. We use θ throughout to denote model parameters (not Θ) and g to denote the reweighting temperature (not Y).

Influence Estimation

Following Koh and Liang,1 the influence of training sample i on the validation loss is

Ii = − giT S, where S = Hθ−1 v

The vector S is computed once per refinement iteration by solving the linear system Hθ S = v and is reused across all N training samples. This factorization reduces the total cost of computing all influence scores from O(N · P2) to O(P2 + N · P)—or, with stochastic solvers, to O(JTP + NP)—since the dominant H −1v solve is performed only once. A positive Ii indicates a training sample that, if removed, would increase validation loss (i.e., beneficial); a negative Ii indicates a potentially harmful sample whose removal would decrease validation loss.

Inverse-Hessian-Vector Solvers

LiSSA for Large Neural Networks

For neural networks, we estimate S = Hθ −1 v using the Linear-time Stochastic Second-order Algorithm (LiSSA).¹² LiSSA exploits the Neumann series expansion, running J independent recursion chains each for T iterations with mini-batch Hessian estimates. Implementation: J = 5 chains, T = 500 iterations, mini-batch size 64, and damping λ = 0.01. Early stopping is applied when the relative change ||St − St ||/ ||St ||< ε = 10-4; convergence was observed within 300–450 iterations in all experiments. Total cost: O(JTP) for the solve, O(NP) for applying S across all training samples. Runtime breakdown across stages (ResNet-18, N = 50,000): H-1v solve ≈31 min (GPU); per-sample dot products ≈52 min (GPU); logging/I/O ≈ 6 min; and total ≈89 min. LiSSA runs on GPU with gradient checkpointing.

Conjugate Gradient for Medium-Scale Models

For logistic regression and shallow networks (P ≤ 105), we use the conjugate gradient (CG) to solve Hθ S = v. CG requires only Hessian-vector products, each computable in O(NP) via a second backward pass. Settings: K = 100 maximum iterations, convergence tolerance ε = 10−5 on residual norm ||Hθ Sk − v ||, and damping λ = 10−3. CG converged in 40–70 iterations for logistic regression and 60–90 for shallow networks. CG runs on CPU (Table 1).

Table 1: Computational complexity of Hessian inversion strategies.
Solver	Time (Solve S)	Time (All Scores)	Memory	Models
LiSSA	O( JTP)	O(NP)	O(P)	Large neural networks
Conjugate gradient	O(KNP)	O(NP)	O(P)	Logistic regression, shallow networks
Exact inversion	O(P3)	O(NP2)	O(P2)	Tiny models (P < 103)
S is computed once; all N influence scores then require O(NP) additional dot products.

Influence Estimation for Tree-Based Models

Gradient Boosted Trees (GBTs)

GBTs optimize a differentiable loss via an additive ensemble, where each tree fits the negative gradient (pseudo-residual) of the current ensemble’s loss. At convergence after M boosting rounds, the pseudo-residual for sample i is

ri (M) = ℓ− ∂ℓ(F  (M−1) (xi), yi)/∂F (M−1) (xi)

We use ri(M) as a surrogate for gi. For cross-entropy loss, the second-order diagonal Hessian entry (Newton weight) is hi = F(xi)(1 − F(xi)), yielding the Newton step ΔFi = ri(M)/hi. The influence approximation is

Local linearization justification: At convergence, the GBT functional F (M) is approximately stable under small perturbations of individual training samples, because each sample influences only a subset of leaf assignments across the M-tree ensemble. For a single sample i with weight ε, the change in the ensemble output can be written as ΔF(x) ≈ ε · (ri(M)/hi) · k(xi, x), where k is a kernel measuring co-leaf membership. This is formally analogous to the quadratic approximation underlying Koh and Liang’s parameter–space influence, applied here to the functional space of additive tree ensembles. The approximation is most accurate when M is large (many boosting rounds) and individual sample weights are small relative to the total ensemble capacity—both conditions that hold in our experimental setting (M = 100 rounds, N ≈ 34,000 training samples).

LOO calibration on a 1,000-sample UCI Adult stratified subset yields Spearman ρ = 0.71 and AUC = 0.73 for harmful-sample detection, compared to ρ = 0.74 and AUC = 0.76 for the leaf-influence baseline¹⁹ on identical splits (Table 3). These results indicate that the leaf-influence method provides modestly better calibration, and practitioners requiring the highest-precision GBT attribution should prefer it; the pseudo-residual approximation offers a practical alternative that integrates into our iterative pipeline.

Random Forests

Random Forests do not optimize a differentiable loss and do not produce pseudo-residuals. We adopt a proximity-based influence approximation following Breiman²¹: the influence of training sample i on validation sample j is estimated as the fraction of trees in which i and j share the same leaf node, weighted by the prediction error on j:

This is a heuristic with two known failure modes: (1) bagging variance—because each tree is trained on a bootstrap sample, two samples may co-occur in a leaf simply due to subsetting rather than genuine similarity, inflating proximity estimates for high-frequency training samples, and (2) leaf-size effects—in shallow trees with large leaves (min_samples_leaf = 2 is a lower bound), many dissimilar samples share a leaf, reducing the discriminative power of proximity.

A failure-case study on UCI Adult (logistic-regression ground truth) showed that the proximity estimator assigns near-zero scores to 18% of genuinely harmful samples (those in the bottom 5th percentile by LOO ground truth), compared to 6% for the GBT pseudo-residual estimator, confirming reduced discriminative power. We therefore recommend Tree SHAP20 or leaf-influence¹⁹ for settings requiring high-precision Random Forest attribution. Performance improvements under proximity-based pruning should be interpreted as suggestive rather than confirmatory.

DistilBERT: Gradient Scope and Bias Handling

For DistilBERT on IMDB, gradients are computed with respect to the classification head parameters only: a linear layer of dimension 768 × 2 with bias, yielding P = 768 × 2 + 2 = 1,538 parameters. The 768-dimensional projection weights account for 1,536 parameters; the 2-dimensional bias vector accounts for the remaining 2. The pre-trained transformer backbone is frozen during fine-tuning and influence computation. Bias parameters are included in the gradient computation.

Head-only influence computation reduces the LOO rank correlation to ρ = 0.76 (vs. ρ = 0.81 for ResNet-18, where the full model is used), reflecting information loss from treating the frozen backbone as a fixed feature extractor. Full-model influence for transformer architectures is computationally challenging due to large P, but methods such as influence sketching⁵ and layer-wise decomposition offer promising directions for improving fidelity without full Hessian computation (Section “DistilBERT Influence Fidelity”).

Dataset Perturbation Strategies

Three perturbation forms are supported. Sample removal eliminates instances with Ii below the α-th percentile (default α = 10, ablated in Section “DCS, Ablations, and Validation Sensitivity”). Label correction flags samples in the bottom 5th percentile for automated relabeling using confident learning.⁹ Sample reweighting assigns wi = σ(g Ii) = 1/(1 + exp(−g Ii)), where g = 1.0 (ablated in Section “DCS, Ablations, and Validation Sensitivity”). Reweighting is preferred over removal in fairness-sensitive or small-dataset settings.

Validation Leakage and Overfitting Risk

Because the validation gradient v = ∇θ L(θ; Dval) is central to all influence scores and is reused across refinement iterations, there is a risk that repeated refinement cycles implicitly overfit to the validation set. We address this risk in four ways. First, validation and test sets are held strictly separate throughout all experiments; influence computation uses Dval and model evaluation uses the held-out test set. Second, we limit refinement to a maximum of three iterations, consistent with the plateau observed in DCS and performance gains.

Third, we report sensitivity analyses in Section “Model Performance After Refinement”, showing DCS and test accuracy as functions of refinement iteration, making any divergence between validation-based pruning decisions and test performance visible. Fourth, we report a holdout-of-holdout sanity check: for CIFAR-10 (10% noise), the validation set is split 50/50 into Dvala (used for influence computation) and DvalB (used to verify that DCS on DvalB tracks DCS on Dvala); the two DCS trajectories differ by at most 0.03 over three iterations, providing no evidence of leakage within the tested range. A full cross-fitting procedure remains an important direction for future work.

Dataset Contribution Score (DCS)

The DCS is defined as:

DCS = (1/N΄) Σi Ĩi, Ĩi = Ii|maxj|Ij|

where N΄ is the number of retained samples and normalization maps scores to [1, +1]. DCS ranges from 1 (all samples harmful) to +1 (all samples maximally beneficial). Important caveats are as follows: (i) aggressive pruning mechanically increases DCS by removing low-influence samples from the average, regardless of downstream performance, and (ii) in imbalanced datasets, minority-class samples carry disproportionately high influence scores, inflating aggregate DCS. Class-stratified DCS is therefore reported by default in all imbalanced experimental settings (Table 6). Table 6 (Section “DCS, Ablations, and Validation Sensitivity”) provides a compact alignment of DCS deltas with downstream performance changes across α thresholds, confirming that DCS is a diagnostic signal and should not be used as a standalone optimization target.

Pipeline Architecture and Pseudocode

The Data Gradient pipeline consists of five stages: (1) Model training with gradient and checkpoint logging; (2) Solve S = Hθ−1 v once using the appropriate solver; (3) Compute Ii = giT S for all training samples; (4) Rank samples and apply perturbation (removal, relabeling, or reweighting); (5) and Retrain on the refined dataset and repeat for up to three iterations. A visual pipeline diagram is provided in Appendix A.

Algorithm 1 (Appendix A) provides the complete pseudocode for the influence computation and refinement loop, including solver selection logic (LiSSA for large models, CG for medium models, and exact inversion for tiny models), convergence monitoring, perturbation thresholds, DCS computation, and early stopping criteria. Full training configurations—optimizer, learning rate schedules, batch sizes, early stopping criteria, and influence hyperparameters—are documented in Appendix B. All parameters are defined at first use in Algorithm 1; acronyms DCS (dataset contribution score) and LiSSA (Linear-time Stochastic Second-order Algorithm) are defined at first occurrence in the text.

Experimental Design

Datasets

For image classification, we use CIFAR-10,²² a 10-class dataset (50,000 training/10,000 test images). Synthetic label noise is introduced by randomly flipping 10% and 20% of training labels following Northcutt et al.⁹ For text classification, we use IMDB,²³ with 25,000 training and 25,000 test reviews. For tabular data, we use UCI Adult,²⁴ with 48,842 instances and 14 features (75% negative class imbalance, ~1–2% duplicate rows). All splits: 70/15/15 (train/val/test), fixed seed 42 (Table 2).

Table 2: Dataset statistics and experimental configurations.
Dataset	Modality	Train	Val	Test	Classes	Noise Condition
CIFAR-10	Image	35,000	7,500	7,500	10	0%, 10%, 20% label flip
IMDB	Text	17,500	3,750	3,750	2	None
UCI Adult	Tabular	34,189	7,326	7,327	2	None (natural noise)
All splits stratified by class; fixed seed 42.

Models

CIFAR-10: ResNet-1825 (PyTorch, batch normalization, no pre-training). IMDB: DistilBERT26 fine-tuned with frozen backbone; gradients computed for classification head only (P = 1,538; see Section “DistilBERT: Gradient Scope and Bias Handling”). UCI Adult: logistic regression, Random Forest (500 trees), XGBoost (100 rounds).

Baselines

Four baselines: Influence Functions1 (exact CG inversion, no pipeline integration); TracIn² (gradient dot products across five checkpoints at epochs 10, 20, 30, 40, 50; checkpoint frequency ablated in Section “TracIn Checkpoint Frequency Ablation”); Representer Point Selection3 (final layer of each neural model); and Random Pruning (same removal count, uniformly random). For GBT and Random Forest settings on UCI Adult, we additionally compare against leaf-influence¹⁹ and Tree SHAP²⁰ as tree-specific baselines on identical splits. All baselines use identical splits, configurations, and evaluation protocols.

Evaluation Protocol

All experiments: five independent random seeds {0, 1, 2, 3, 42}, weights re-initialized, and data reshuffled. Results: mean ± standard deviation. Statistical tests: paired two-sided Wilcoxon signed-rank test (­nonparametric, appropriate for n = 5 seeds) with α = 0.05. Effect sizes: Cohen’s d computed from per-seed values. Multiple-comparison correction: Benjamini–Hochberg FDR at q = 0.05 across all dataset–baseline pairs within each model family. Primary metrics: classification accuracy and macro-averaged F1-score on the held-out test set. Secondary metrics: DCS before and after refinement, mislabel detection precision and recall, influence computation time, and peak memory.

Leave-One-Out Calibration

LOO validation on stratified 1,000-sample subsets of each dataset. For each sample, the model is retrained from scratch with that sample excluded and the change in validation loss recorded. Calibration metrics: Spearman rank correlation ρ, mean absolute error (MAE), and AUC for harmful-sample detection (where ground truth is the synthetically flipped labels for CIFAR-10 noisy variants). Direct comparison between our pseudo-residual GBT estimator and the leaf-influence baseline19 uses rank correlation and AUC confidence intervals on identical LOO splits.

Ablation Studies

Four ablation experiments: (i) removal threshold α ∈ {5, 10, 15, 20} percentile; (ii) LiSSA recursion depth T ∈ {100, 300, 500, 1000}—LOO rank correlation versus computation time; (iii) reweighting temperature g ∈ {0.5, 1.0, 2.0, 5.0}; and (iv) TracIn checkpoint frequency (1, 2, 5, 10 checkpoints)—mislabel detection precision versus storage cost. All ablation figures are presented in graphical form in Appendix C.

Results

Leave-One-Out Calibration

Table 3 reports the Spearman rank correlation, MAE, and AUC for harmful-sample detection against LOO ground truth. Correlations are the highest for logistic regression (ρ = 0.93) and the lowest for Random Forests (ρ = 0.58). For CIFAR-10 with 10% synthetic noise, the data gradient estimator achieves AUC = 0.79 ± 0.03 for detecting flipped labels, compared to AUC = 0.61 ± 0.05 for the proximity-based Random Forest estimator. The GBT pseudo-residual estimator achieves ρ = 0.71 and AUC = 0.73, compared to ρ = 0.74 and AUC = 0.76 for the leaf-influence baseline on identical splits, indicating that the tree-specific method provides modestly better calibration.

Table 3: LOO calibration: Spearman ρ, MAE, and AUC for harmful-sample detection. Mean ± std across five seeds.
Dataset	Model	Spearman ρ	MAE	AUC (Harmful)	Solver	Notes
CIFAR-10	ResNet-18	0.81 ± 0.03	0.014 ± 0.002	0.79 ± 0.03	LiSSA	Head+backbone
IMDB	DistilBERT	0.76 ± 0.04	0.018 ± 0.003	0.74 ± 0.04	LiSSA	Head only (P = 1,538)
UCI Adult	Logistic Reg.	0.93 ± 0.02	0.006 ± 0.001	0.88 ± 0.02	CG	Full model
UCI Adult	XGBoost (ours)	0.71 ± 0.05	0.021 ± 0.004	0.73 ± 0.04	Pseudo-res.	vs. leaf-inf.: ρ = 0.74
UCI Adult	RF (ours)	0.58 ± 0.07	0.034 ± 0.006	0.61 ± 0.05	Proximity	Heuristic; see Sect. “Random Forests”
UCI Adult	XGBoost (leaf-inf.)19	0.74 ± 0.04	0.019 ± 0.003	0.76 ± 0.04	Leaf-inf.	Tree-specific baseline
UCI Adult	RF (Tree SHAP)20	0.67 ± 0.05	0.026 ± 0.004	0.69 ± 0.04	Tree SHAP	Tree-specific baseline
AUC computed against synthetically flipped labels on CIFAR-10 (10% noise); against influence-ranked removal ground truth on UCI Adult. Confidence intervals on GBT vs. leaf-influence AUC overlap at the 95% level.

Mislabel Detection (Table 4)

Table 4: Mislabel detection on CIFAR-10. Mean ± std across 5 seeds.
Method	10% Prec.	10% Rec.	10% AUC	20% Prec.	20% Rec.	20% AUC	Ref.
Data gradient	0.71 ± 0.04	0.64 ± 0.05	0.79 ± 0.03	0.63 ± 0.05	0.58 ± 0.06	0.71 ± 0.04	(ours)
Influence Func.	0.70 ± 0.04	0.63 ± 0.05	0.78 ± 0.03	0.62 ± 0.05	0.57 ± 0.06	0.70 ± 0.04	[1]
TracIn	0.69 ± 0.05	0.61 ± 0.06	0.76 ± 0.04	0.60 ± 0.06	0.55 ± 0.07	0.68 ± 0.05	[2]
Representer Pts.	0.65 ± 0.05	0.58 ± 0.06	0.72 ± 0.04	0.57 ± 0.06	0.51 ± 0.07	0.64 ± 0.05	[3]
Random Pruning	0.10	0.10	0.50	0.20	0.20	0.50	—
AUC against synthetically flipped label ground truth. Gains confirm influence-guided selection over random pruning.

Model Performance After Refinement

Table 5 reports the test accuracy and macro F1-score with effect sizes (Cohen’s d) and 95% confidence intervals derived from per-seed distributions. All significance tests are Wilcoxon signed-rank with Benjamini–Hochberg FDR correction (q = 0.05). Random Pruning produces no consistent improvement, confirming that gains are attributable to sample selection quality rather than dataset size reduction. The data gradient framework matches or outperforms all baselines across settings. Gains for Random Forest are modest and do not survive FDR correction, consistent with the lower LOO calibration reported in Section “Leave-One-Out Calibration.”

Table 5: Test performance before and after refinement. Mean ± std across 5 seeds.
Dataset/Model	Metric	Unrefined	Rand. Prune	Inf. Func.1	TracIn2	Representer3	Data gradient	Cohen’s d	95% CI
CIFAR-10 0%/ResNet-18	Acc.	87.2 ± 0.4	87.1 ± 0.5	88.6 ± 0.4*	88.4 ± 0.5*	88.0 ± 0.5*	89.5 ± 0.3**	1.12	[0.71,1.53]
CIFAR-10 10%/ResNet-18	Acc.	83.1 ± 0.6	83.3 ± 0.6	85.2 ± 0.5*	85.0 ± 0.5*	84.5 ± 0.6*	86.4 ± 0.4**	0.98	[0.59,1.37]
IMDB/DistilBERT	F1	0.841 ± 0.006	0.843 ± 0.007	0.858 ± 0.006*	0.856 ± 0.007*	0.851 ± 0.007*	0.872 ± 0.005**	0.87	[0.49,1.25]
UCI adult/Log. Reg.	Acc.	84.3 ± 0.5	84.2 ± 0.6	85.1 ± 0.5*	84.9 ± 0.5	84.7 ± 0.6	85.6 ± 0.4**	0.72	[0.35,1.09]
UCI adult/XGBoost	Acc.	86.7 ± 0.4	86.6 ± 0.5	87.2 ± 0.5*	87.0 ± 0.5	86.9 ± 0.5	87.5 ± 0.4**	0.41	[0.06,0.76]
UCI adult/Rand. Forest	Acc.	85.9 ± 0.5	85.8 ± 0.6	86.1 ± 0.5	86.0 ± 0.5	85.9 ± 0.5	86.3 ± 0.5	0.21	[−0.14,0.56]
*p < 0.05 vs. unrefined (Wilcoxon, BH-corrected); **p < 0.05 vs. Random pruning (Wilcoxon, BH-corrected). Cohen’s d and 95% CI from Data Gradient vs. Unrefined per-seed pairs. Random Forest gains do not survive FDR correction.

DCS, Ablations, and Validation Sensitivity

LiSSA ablation: LOO rank correlation improves from ρ = 0.61 (T = 100) to ρ = 0.80 (T = 300), plateauing at ρ = 0.81 (T = 500 and T = 1000), while computation time scales linearly; T = 500 is selected as the optimal trade-off. Reweighting: stable across g [ {0.5, 1.0, 2.0}, degrades at g = 5.0 (binary-like behavior approaching hard removal). DCS and test accuracy track each other through three refinement iterations with no evidence of divergence. The holdout-of-holdout sanity check (Section “Validation Leakage and Overfitting Risk”) confirms no leakage signal within this range (Table 6).

Table 6: DCS before and after refinement with DCS–performance alignment across thresholds. Mean ± std, 5 seeds.
Dataset	Model	DCS Before	DCS After (á = 10)	ΔDCS	DCS (á = 20)	Test Acc. (á = 20)	Notes
CIFAR-10 (0%)	ResNet-18	0.31 ± 0.02	0.47 ± 0.02	+0.16	0.58 ± 0.02	88.9 ± 0.4
CIFAR-10 (10%)	ResNet-18	0.18 ± 0.03	0.41 ± 0.03	+0.23	0.54 ± 0.03	85.7 ± 0.5
IMDB	DistilBERT	0.38 ± 0.02	0.52 ± 0.02	+0.14	0.61 ± 0.02	0.868 ± 0.006
UCI adult	Logistic Reg.	0.43 ± 0.03	0.58 ± 0.02	+0.15	0.66 ± 0.02	85.1 ± 0.5
UCI adult	XGBoost	0.46 ± 0.02	0.61 ± 0.02	+0.15	0.68 ± 0.02	87.1 ± 0.4
UCI adult (stratified)	Logistic Reg.	0.43 ± 0.03	0.57 ± 0.02	+0.14	—	—	Class-strat.
The a = 20 column illustrates DCS–performance divergence: DCS rises, while test accuracy declines toward baseline, confirming that DCS is diagnostic, not an optimization target. Stratified DCS reported within class to control for class imbalance.

TracIn Checkpoint Frequency Ablation (Table 7)

Table 7: TracIn checkpoint frequency ablation on CIFAR-10 (10% noise). Mean ± std, 5 seeds.
Checkpoints	Precision (10%)	Recall (10%)	Storage Overhead	Note
1 (final only)	0.61 ± 0.06	0.54 ± 0.07	1×	Equivalent to single gradient dot product
2	0.65 ± 0.05	0.58 ± 0.06	2×
5 (default)	0.69 ± 0.05	0.61 ± 0.06	5×	Default in our experiments
10	0.70 ± 0.05	0.62 ± 0.06	10×	Marginal gain over 5
Precision and recall improve substantially from 1 to 5 checkpoints; gains plateau beyond 5. Default of five checkpoints represents a reasonable cost–performance trade-off.

Scalability (Table 8)

Table 8: Scalability results (ResNet-18, CIFAR variants, LiSSA T = 500). Mean ± std, 3 runs.
Dataset Size (N)	Influence Time	Training Time	Overhead Ratio	Peak Mem. (GPU)	Hardware
10,000	18 ± 1 min	42 ± 2 min	0.43×	8.2 ± 0.3 GB	V100 32GB
25,000	43 ± 2 min	98 ± 3 min	0.44×	14.1 ± 0.4 GB	V100 32GB
50,000	89 ± 3 min	201 ± 5 min	0.44×	24.3 ± 0.6 GB	V100 32GB
100,000	178 ± 5 min	407 ± 8 min	0.44×	31.7 ± 0.8 GB	V100 32GB
The overhead ratio is stable at ≈0.44× across all scales, consistent with O(NP) complexity. Validated range: up to 100,000 samples on the hardware above. GPU portability: LiSSA requires a CUDA-capable GPU with at least 16 GB VRAM for N ≥ 50,000; CG runs on CPU for all logistic regression and tree models.

Discussion

Interpreting the Results

Three main conclusions follow from the experimental results. First, influence-guided refinement produces consistent, statistically significant improvements over unrefined baselines across all differentiable model families, with effect sizes ranging from Cohen’s d = 0.41 to 1.12. Gains are modest under clean label conditions but increase substantially under label noise, where mislabeled sample identification drives performance. This pattern is consistent with Koh and Liang,1 and Northcutt et al.⁹

Second, the comparison against TracIn,² Representer Point Selection,³ and tree-specific methods^19,20 reveals broadly comparable detection performance, with the primary practical advantage of the data gradient framework lying in pipeline integration. Third, LOO calibration and AUC results confirm that Random Forest influence estimates are substantially less reliable than differentiable-model estimates, and the modest gains for Random Forest do not survive FDR correction.

DCS: Properties, Caveats, and Guidance

DCS reliably tracks the direction of performance improvement across refinement iterations in most settings but diverges from test accuracy when pruning is aggressive. The α = 20 ablation (Table 6) makes this visible: DCS continues rising, while test accuracy declines toward baseline. Practitioners should treat DCS as a complementary signal—diagnostic, not an optimization target. Class-stratified DCS is recommended by default in imbalanced settings. When minority classes are highly influential, aggregate DCS will be dominated by minority-class samples even if majority-class quality is poor; stratification makes this visible. The threshold alignment in Table 6 provides actionable guidance: DCS gains above α = 10 no longer correlate with test performance gains, and practitioners should not exceed this threshold without explicit holdout verification.

DistilBERT Influence Fidelity

Computing influence scores only for the classification head (P = 1,538 parameters) reduces LOO rank correlation to ρ = 0.76 compared to ρ = 0.81 for ResNet-18. This gap reflects the information loss from treating the frozen transformer backbone as a fixed feature extractor. Full-model influence for transformer architectures is computationally challenging due to large P, but methods such as influence sketching⁵ and layer-wise decomposition offer promising directions for improving fidelity without full Hessian computation.

Validation Leakage

The sensitivity analysis confirms that DCS and test accuracy track each other through three refinement iterations with no evidence of divergence. The holdout-of-holdout sanity check (Section “Validation Leakage and Overfitting Risk”) further corroborates this. However, this analysis cannot rule out leakage over more iterations or in settings where validation and test distributions differ. We recommend the three-­iteration limit as a practical safeguard. A full cross-fitting procedure—rotating Dval across folds—remains the most important near-term methodological improvement for reducing leakage risk.

Fairness Audit

Group-wise influence score distributions were computed on UCI Adult (logistic regression) stratified by sex and race. Female-coded samples had mean normalized influence score 0.28 ± 0.04 versus 0.34 ± 0.03 for male-coded samples (Wilcoxon p < 0.01), meaning that a standard α = 10 threshold would remove proportionally more female-coded samples. Group-aware pruning—applying α = 10 separately within each demographic stratum—preserved group representation while maintaining 85.3 ± 0.4% test accuracy (vs. 85.6 ± 0.4% for standard pruning). Reweighting without removal preserved both accuracy (85.4 ± 0.4%) and group representation fully. These results confirm that group-aware strategies are preferable in fairness-sensitive deployment contexts, and reweighting is the recommended default when demographic balance is a constraint.

Limitations

The influence function estimator assumes model convergence and a locally quadratic loss surface, both imperfect for overparameterized neural networks. For DistilBERT, influence is computed for the head only, limiting fidelity. The iterative refinement loop assumes full retraining feasibility, which does not hold for large foundation models. Experimental validation uses benchmark datasets with synthetic noise; naturalistic noise patterns may yield different calibration. The Random Forest proximity estimator is a heuristic; practitioners requiring high-precision attribution should use Tree SHAP20 or leaf-influence.¹⁹

Future Directions

Incremental influence estimation—updating scores as new data arrive without full recomputation¹³ would enable real-time monitoring in production pipelines. Integration with AutoML could allow influence scores to inform feature selection and curriculum design. For foundation models, influence sketching and distributed Hessian approximation methods warrant investigation. Full-model influence for fine-tuned transformers and cross-fitting-based validation strategies are the most immediately impactful near-term extensions.

Conclusion

This paper introduced the data gradient framework for per-sample influence estimation in end-to-end machine learning pipelines. The framework computes Ii = −giT S, where S = Hθ −1 Dθ Lval is solved once per refinement iteration—using LiSSA for large neural networks and CG for smaller models—and reused across all training samples. Influence scores drive iterative dataset refinement through sample removal, label correction, or reweighting. The dataset contribution score (DCS) tracks aggregate dataset quality across iterations.

Experiments across image, text, and tabular modalities demonstrated statistically significant improvements, with effect sizes ranging from Cohen’s d = 0.41 to 1.12, confirmed by Wilcoxon tests with Benjamini–Hochberg correction. LOO calibration confirmed influence estimate reliability for differentiable models and quantified the limitations of tree-based approximations. Scalability was validated up to 100,000 samples with consistent 0.44× training time overhead.

Four findings carry implications beyond this framework. First, the once-computed S vector substantially reduces influence computation cost relative to naive per-sample Hessian solves. Second, DCS diverges from test performance under aggressive pruning, cautioning against its use as a standalone optimization target. Third, fairness-aware pruning strategies preserve group representation with minimal accuracy cost. Fourth, the DistilBERT head-only gradient scope reduces LOO fidelity, motivating full-model influence methods for transformers. The data gradient framework contributes a practically deployable, empirically validated, and methodologically honest tool to the data-centric machine learning toolkit.

References

1. Koh PW, Liang P. Understanding black-box predictions via influence functions. In: Precup D, Teh YW, eds. Proceedings of the 34th International Conference on Machine Learning. Vol. 70. PMLR; 2017:1885–1894.
2. Pruthi G, Liu F, Kale S, Sundararajan M. Estimating training data influence by tracing gradient descent. Adv Neural Inf Process Syst. 2020;33:19920–19930.
3. Yeh CK, Kim J, Yen IEH, Ravikumar PK. Representer point selection for explaining deep neural networks. Adv Neural Inf Process Syst. 2018;31:9291–9301.
4. Swayamdipta S, Schwartz R, Lourie N, et al. Dataset cartography: mapping and diagnosing datasets with training dynamics. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2020:9275–9293. https://doi.org/10.18653/v1/2020.emnlp-main.746
5. Guo C, Pleiss G, Sun Y, Weinberger KQ. FastIF: scalable influence functions for efficient model interpretation and debugging. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2021:10333–10350. https://doi.org/10.18653/v1/2021.emnlp-main.808
6. Park S, Georgiev K, Ilyas A, Leclerc G, Madry A. TRAK: attributing model behavior at scale. In: Krause A, Brunskill E, Cho K, Engelhardt B, Sabato S, Scarlett J, eds. Proceedings of the 40th International Conference on Machine Learning. Vol. 202. PMLR; 2023:27074–27113.
7. Bae J, Ng N, Lo A, Ghassemi M, Grosse RB. If influence functions are the answer, then what is the question? Adv Neural Inf Process Syst. 2022;35:17953–17967.
8. Sambasivan N, Kapania S, Highfill H, Akrong D, Paritosh P, Aroyo LM. “Everyone wants to do the model work, not the data work”: data cascades in high-stakes AI. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM; 2021:1–15. https://doi.org/10.1145/3411764.3445518
9. Northcutt CG, Jiang L, Chuang I. Confident learning: estimating uncertainty in dataset labels. J Artif Intell Res. 2021;70:1373–1401. https://doi.org/10.1613/jair.1.12125
10. Zha D, Bhat ZP, Lai KH, et al. Data-centric artificial intelligence: a survey. ACM Comput Surv. 2023;56(7):1–36. https://doi.org/10.1145/3611703
11. Jakubik J, Vussow M, Hemmer P, Pretschner A, Kuhl N. Data-centric artificial intelligence. Bus Inf Syst Eng. 2024;66(1):5–20.
    https://doi.org/10.1007/s12599-023-00824-5
12. Agarwal N, Bullins B, Hazan E. Second-order stochastic optimization for machine learning in linear time. J Mach Learn Res. 2017;18(116):1–40.
13. Saini R, Mussbacher G, Guo JL, Kienzle J. Machine learning-based incremental learning in interactive domain modelling. In: Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems. 2022:176–186.
14. Ghorbani A, Zou J. Data Shapley: equitable valuation of data for machine learning. In: Chaudhuri K, Salakhutdinov R, eds. Proceedings of the 36th International Conference on Machine Learning. Vol. 97. PMLR; 2019:2242–2251.
15. Jia R, Dao D, Wang B, et al. Towards efficient data valuation based on the Shapley value. In: Chaudhuri K, Sugiyama M, eds. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics. Vol. 89. PMLR; 2019:1167–1176.
16. Loizou A, Tsoumakos D. Chunked data Shapley: A scalable dataset quality assessment for machine learning. In: Proceedings of the 34th ACM International Conference on Information and Knowledge Management. 2025:1958–1967.
17. Sundararajan M, Taly A, Yan Q. Axiomatic attribution for deep networks. In: Precup D, Teh YW, eds. Proceedings of the 34th International Conference on Machine Learning. Vol. 70. PMLR; 2017:3319–3328.
18. Shrikumar A, Greenside P, Kundaje A. Learning important features through propagating activation differences. In: Precup D, Teh YW, eds. Proceedings of the 34th International Conference on Machine Learning. Vol. 70. PMLR; 2017:3145–3153.
19. Sharchilev B, Ustinovskiy Y, Serdyuk A, de Rijke M. Finding influential training samples for gradient boosted decision trees. In: Dy J, Krause A, eds. Proceedings of the 35th International Conference on Machine Learning. Vol. 80. PMLR; 2018:4577–4585.
20. Lundberg SM, Erion G, Chen H, et al. From local explanations to global understanding with explainable AI for trees. Nat Mach Intell. 2020;2(1):56–67. https://doi.org/10.1038/s42256-019-0138-9
21. Breiman L. Random forests. Mach Learn. 2001;45(1):5–32. https://doi.org/10.1023/A:1010933404324
22. Krizhevsky A. Learning Multiple Layers of Features From Tiny Images (Technical Report). University of Toronto; 2009.
23. Maas AL, Daly RE, Pham PT, Huang D, Ng AY, Potts C. Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2011:142–150.
24. Dua D, Graff C. UCI Machine Learning Repository [Data set]. University of California, Irvine, School of Information and Computer Sciences; 2017. https://archive.ics.uci.edu/ml
25. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE; 2016:770–778.
26. Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv. 2019. https://doi.org/10.48550/arXiv.1910.01108

Appendix

Cite this article as:
Kamene K. The Data Gradient: Quantifying Dataset Influence in End-to-End Machine Learning Systems. Premier Journal of Computer Science 2026;6:100014

Export to EndNote Export to Mendeley