Skip to content
AIBRAINWORX / OpenSourceCoder.comLOCAL-FIRST. BUILT IN THE OPEN.
Infrastructure & Data / Text & Coding LLMs

Dharma-AI Shows Pre-Dispatch CatBoost Outperforms FLOP Math for Fine-Tuning Time

Analyzing 272 production SFT runs on NVIDIA H200 clusters, Dharma-AI reports that a 24-feature CatBoost model cuts duration prediction error to 1.71 hours MAE where FLOP formulas miss by over 12 hours.

Listen to this articleBrowser Voice
Ready
Official company logo or product image for dharma-ai-predicting-fine-tuning-time-catboost-h200
Official image or organization logo from huggingface.co; identification only, no endorsement.
Event / context: 2026-11-18. Source check: October 28, 2026.

Why FLOP estimates miss production fine-tuning reality

On November 18, 2026, researchers at Dharma-AI—Bruno Duarte, Gabriel Pimenta de Freitas Cardoso, Isis F Carvalho, Breno de Almeida Beleza, and Felipe Lima—published a technical report on Hugging Face previewing their upcoming arXiv paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The team examined a persistent scheduling bottleneck on shared GPU clusters: teams routinely estimate supervised fine-tuning (SFT) duration using the closed-form formula T_theoretical = (6 · P · D) / (N_GPU · FLOPS_peak · MFU), only to see jobs finish hours early or slip into the next day.

To quantify that discrepancy in a production setting, Dharma-AI analyzed 272 real SFT runs representing roughly 7,382 GPU-hours from a corporate cluster. According to the team, the compute-only FLOP baseline explained little of the actual wall-clock variance (R² = 0.31 at best, with a 12.41-hour mean absolute error), whereas a CatBoost regressor restricted strictly to pre-dispatch configuration metadata achieved R² = 0.98 and a 1.71-hour mean absolute error in five-fold cross-validation.

Pre-dispatch architecture and engineered overhead signals

The dataset spans jobs executed on 1, 2, or 4 NVIDIA H200 GPUs drawn from a shared pool of 8 to 32 H200s across 1 to 4 nodes, running concurrently alongside other training and inference workloads. Workloads covered several model families up to 15 billion parameters, mixing LoRA and full fine-tuning across Hugging Face Transformers and Unsloth. The researchers modeled the software stack as a categorical variable rather than assuming a fixed speedup between the two frameworks. Measured end-to-end times—defined from GPU allocation to job completion, excluding queue wait time—ranged from 1.6 to 172.4 hours, with a mean of 23.43 hours and a median of 15.29 hours.

Dharma-AI notes that closed-form FLOP math captures only the core optimization loop while ignoring operational overheads that vary nonlinearly with job configuration and cluster contention. Before step one, jobs incur environment initialization, model downloads, weight loading, and tokenization; during execution, periodic evaluations, checkpoint saves, and final uploads to object storage compete with neighboring tenants for network and storage I/O. To expose these costs before a job launches, the team engineered derived features including num_eval_data (total samples evaluated across all cycles), num_saves and num_params_saved (checkpoint I/O volume), per-GPU token counts, and baseline_time (the FLOP estimate computed with MFU fixed at 0.45). Using correlation filtering, mutual information, permutation importance across 18 candidate feature sets, and MAE-targeted grid search, they distilled the final CatBoost predictor down to 24 pre-dispatch features.

Benchmarking CatBoost against Calculon and MFU baselines

In stratified five-fold cross-validation, Dharma-AI compared analytical baselines against linear and gradient-boosted regressors. Even after adding scheduled evaluation tokens to the FLOP count and sweeping Model FLOPs Utilization (MFU) from 0.25 to 0.65, no static MFU value resolved the gap: at 30% MFU, the FLOP baseline yielded R² = 0.31, RMSE = 20.45 h, and MAE = 12.41 h; at 45% MFU, R² fell to 0.18 (MAE = 13.01 h); and at 60% MFU, R² dropped to 0.01 (MAE = 14.62 h).

The authors also tested Calculon (Isaev et al., SC '23), an analytical layer-graph performance model extended by the team to support LoRA, grouped-query attention, fused attention, bfloat16, and SwiGLU. Calculon posted R² = -0.51, RMSE = 30.40 h, and MAE = 20.36 h, estimating a median of just 11% of observed wall-clock duration—demonstrating how much production time occurs outside step-level compute. Linear Regression (R² = 0.69 ± 0.14, MAE = 7.52 ± 0.49 h) and Ridge regression (R² = 0.73 ± 0.09, MAE = 7.23 ± 0.68 h) improved substantially over analytical formulas but stalled on nonlinear interactions. Tree ensembles closed the gap: XGBoost reached R² = 0.97 ± 0.01 (RMSE = 4.18 ± 1.17 h, MAE = 1.95 ± 0.32 h), while CatBoost led with R² = 0.98 ± 0.01, RMSE = 3.52 ± 0.78 h, MAE = 1.71 ± 0.33 h, and a median error of 0.51 hours. Feature-importance rankings (PredictionValuesChange) and SHAP analysis placed num_eval_data first, followed by num_trainable_params, batch_eval_size_percent, num_all_params, num_gpus*batch_train_percent, and num_params_saved.

Scheduling impact on shared GPU clusters

Because Dharma-AI's predictor uses zero runtime logs or partial progress metrics, cluster orchestrators and MLOps engineers can evaluate job duration at submission time. On multi-tenant H200 pools, tighter pre-dispatch estimates help schedulers pack 1-, 2-, and 4-GPU allocations more efficiently, reducing stranded capacity when jobs finish early and curbing cascading queue slips when jobs run long. For practitioners using Transformers or Unsloth, the feature-importance breakdown also serves as an operational reminder: aggressive evaluation cadences, large evaluation splits, and frequent checkpoint uploads can influence total wall-clock cost nearly as much as parameter scale.

Scope boundaries, caveats, and upcoming paper details

All accuracy figures come from Dharma-AI's internal five-fold cross-validation on one corporate environment rather than independent multi-site benchmarks, and the authors explicitly caution that feature importance does not establish causality. The training corpus is bounded to SFT workloads on models up to 15 billion parameters running across 1 to 4 nodes of H200 GPUs with Transformers and Unsloth, and excludes scheduler queue wait time. Models trained on one cluster's object-storage throughput and multi-tenant contention profile may require recalibration before generalizing to different hardware generations, interconnects, or pre-training regimes. Finally, the Hugging Face post serves as a pre-arXiv preview, and public code weights or open-source licensing terms for the predictor had not yet been detailed in the announcement.

Automated Vertex AI (gemini-4-hydrogen) primary-source report; reviewed 2026-11-19.

Follow your curiosity.

Pass the signal.