Synthetic Data Distillation Token Yield & Cost Calculator

Synthetic Data ROI: Token Yield, Quality & When It Beats Real Data

Training on synthetic data -- model-generated examples instead of human-labeled ones -- went from research curiosity to standard practice. The economics: human labeling costs $0.05-$5 per example; synthetic generation costs $0.001-$0.05. A 100k-example dataset costs $500k human-labeled vs $2k synthetic. But the yield question is quality-adjusted: how many synthetic tokens equal one human token for model improvement? Estimates range 3:1 to 10:1 depending on task -- you need more synthetic data, but it is 100x cheaper, so the ROI still crushes.

The failure mode is model collapse -- training on your own outputs recursively degrades quality (the photocopy-of-a-photocopy problem). Mitigations that work: mix synthetic with real data (30-70% synthetic is the common safe band), filter aggressively (keep top 20-40% by quality scoring -- generation is cheap, curation is the work), use a stronger teacher model than the student, and validate on held-out human data. The tasks where synthetic shines: instruction tuning (diverse prompts), reasoning traces (teacher chain-of-thought), code (verifiable correctness), and low-resource languages (where human data barely exists). Where it struggles: tasks needing genuine human judgment, taste, or novelty.

Token yield math: effective_tokens = synthetic_tokens x quality_ratio x diversity_factor. A 10M-token synthetic set at 5:1 quality ratio with 0.7 diversity = 1.4M effective tokens. Compare against the human-data cost for 1.4M equivalent tokens to get true ROI. The calculator models generation cost, filtering yield, quality ratios by task type, and the real-data baseline -- because synthetic data decisions should be arithmetic, not vibes.

The frontier is shifting: frontier labs now train heavily on synthetic reasoning data (RL-verified traces), and the quality ratios keep improving as teacher models get stronger. But the curation bottleneck grows with it -- generating 100M tokens is cheap; verifying which 20M are good is the expensive human step. Invest in eval harnesses before generation scale: a bad synthetic set at 10M tokens poisons more than a good set at 1M helps. Measure twice, generate once.