Fitting Scaling Laws from Your Own Runs

The Problem

You have just finished a sweep. Twenty models, different widths and depths, different token budgets, all trained on the same corpus. The loss-versus-FLOPs plot is noisy but clearly slopes downward. Now someone asks the only question that matters:

"We have 10^{21} FLOPs. Should we train an 8B model or a 14B model, and for how many tokens?"

Answering from a paper figure is dangerous. Published scaling laws were fitted on someone else's data, someone else's optimizer, someone else's learning-rate schedule, and often someone else's token distribution. Their constants are not your constants. Worse, the wrong functional form can silently invert the recommendation: a model that looks better under a Kaplan fit can look worse under a Chinchilla fit, because the two curves disagree on how fast the irreducible loss floor approaches.

The only safe answer is to fit the law yourself, on your own runs, and to understand exactly where the fit is fragile.

The Concept

Power laws and why the log plot lies to you

The simplest scaling law is a power law in one variable:

Take logs and it becomes a straight line:

That linearity is why a log-log plot is the first thing everyone draws. But the straight line is only straight if the model has no irreducible loss floor. Real language models do. The Chinchilla form separates the floor from the learnable part:

where E is the irreducible loss, A and B are scale constants, and \alpha \approx 0.34, \beta \approx 0.28 are the famous Hoffmann exponents. If you log the full L, you get a curve that bends toward \log E at large scale. If you log (L - E), you recover a linear combination of log-power terms.

This means estimating E is not optional. Treat it as a nuisance parameter and the fitted exponent will absorb the floor, usually flattening the slope and underestimating \alpha.

Kaplan vs. Chinchilla: the same data, different conclusions

Kaplan et al. (2020) fitted models trained for a fixed number of steps. As N grew, D grew with it, and the learning-rate schedule was not re-tuned per model size. The result was a parameter-scaling exponent \alpha_N \approx 0.076 — much smaller than the value Chinchilla later found.

The difference is not academic. A smaller exponent says "build a bigger model"; a larger exponent says "the model-size term saturates quickly, so you also need more data." Kaplan's data was contaminated by three things:

1. Covariation. N and D moved together, so the two power terms were confounded.
2. Learning-rate schedule. Small and large models received schedules that were not individually optimal, adding size-dependent noise.
3. Functional form. The Kaplan law did not include a clean irreducible-loss term, so the floor bled into the exponent.

Hoffmann et al. (2021) fixed all three. They trained families where either N or D was held fixed, tuned learning rates per size, and explicitly modeled E. The resulting exponents (\alpha \approx 0.34, \beta \approx 0.28) give the D \approx 20N compute-optimal rule.

The three Chinchilla estimation methods

Chinchilla did not rely on a single fit. It cross-checked three experimental designs, each with a different blind spot:

Method 1 — fixed N, sweep D. Train one model size on many token counts. You fit:

The term in parentheses is a constant offset. This method cleanly estimates \beta and B, but only if N is large enough that the A/N^\alpha term has stopped changing rapidly. If N is too small, the offset still contains a shrinking N term and \beta is biased.

Method 2 — fixed D, sweep N. Train many model sizes on the same token count. You fit:

This cleanly estimates \alpha and A, with the same caveat: D must be large enough that the data term is nearly flat.

Method 3 — joint fit L(N, D). Collect points across the (N, D) plane and fit all five parameters at once:

This is the most data-efficient when the sweep covers both axes, but the parameters are coupled. A bad estimate of E pulls \alpha and \beta; a sparse grid can create multiple local minima with similar loss.

The right workflow is to use Method 3 as the headline, then sanity-check \alpha against Method 2 and \beta against Method 1. If they disagree, one of your experimental axes is contaminated.

Practical fitting traps

Data-layout contamination. If every run uses the same batch order, the same evaluation split, or the same seed initialization pattern, the residuals are correlated. Bootstrap confidence intervals will look narrower than reality because you are resampling correlated noise.

Estimating E. The irreducible loss is the value the curve approaches, not the lowest loss you happened to observe. Estimating E from the minimum of a small sweep usually overestimates it and steepens the fitted curve. A safer approach is to fit E jointly and check that the fitted E is close to the loss of your largest, longest run.

Small-model learning rates. If the learning rate was not tuned for the smallest models, their losses sit above the true scaling curve. Because small models are the most informative about the large-N asymptote, this biases \alpha downward.

Confidence intervals from bootstrap. Bootstrap percentiles work only when the design matrix is well-conditioned and the noise is independent. If your sweep is small or the points cluster along a diagonal in (N, D) space, the CI is optimistic. Report it, but do not treat it as a guarantee.

Build It

code/main.py generates a synthetic sweep, fits the full Chinchilla law, fits the two single-variable laws, bootstraps confidence intervals, and solves for compute-optimal (N^*, D^*).

The synthetic sweep. Thirty (N, D, L) points are drawn from the known law L = E + A/N^\alpha + B/D^\beta plus Gaussian noise. This is your stand-in for a real hyperparameter sweep exported from a tracker.

Fitting. For fixed exponents (\alpha, \beta), the coefficients (E, A, B) are solved by ordinary least squares. The nonlinear exponents are then optimized by gradient descent on the reduced squared-error objective. The gradient is analytical thanks to the normal equations: X^T r = 0 at the optimal linear coefficients, so only the exponents need iterative optimization. Multi-restarts protect against local minima.

Single-variable checks. Two additional synthetic datasets hold one axis fixed — fixed D with varying N, and fixed N with varying D — and are fitted with L(X) = E + A/X^\alpha. These recover the true exponents when the experimental design is clean.

Bootstrap CI. The 30 points are resampled with replacement 100 times; the full model is refit on each resample; the 2.5th and 97.5th percentiles of the exponents form a 95% confidence interval.

Compute-optimal frontier. For a budget C \approx 6ND, the optimal allocation has a closed form:

The script prints the frontier from 10^{19} to 10^{23} FLOPs and answers the 8B-vs-14B question for 10^{21} FLOPs.

Run it:

consoleCopypython3 main.py

Verify It

code/tests/test_main.py locks down the behavior that makes the fit trustworthy:

• generated synthetic data has the expected shape and stays above the irreducible floor;
• the full Chinchilla fit recovers the true \alpha and \beta within 5%;
• a fixed-D single-variable fit recovers \alpha;
• a fixed-N single-variable fit recovers \beta;
• the closed-form compute-optimal (N^*, D^*) agrees with a brute-force grid search;
• the Kaplan-style no-E fit is systematically biased away from the true exponent;
• bootstrap confidence intervals contain the true exponents;
• optimal model size grows monotonically with compute budget.

Why This Matters for Engineers

• The curve is a budget decision. "8B or 14B" is exactly the question a fitted scaling law answers. Fitting your own data removes the risk of applying someone else's constants to your corpus.
• The functional form matters. Ignoring E or conflating N and D can flip the recommendation. Method 3 plus Method 1/2 cross-checks is the minimum due diligence.
• Learning-rate schedules are part of the experiment. If small models are under-trained relative to their optimal schedule, the fitted exponent is wrong and your budget forecast is wrong.
• Confidence intervals are honesty checks. A tight CI from a clean sweep gives you license to interpolate. A wide CI tells you to run more experiments before committing silicon.
• Extrapolation is dangerous. Scaling laws are reliable interpolators and risky extrapolators. Fit inside the envelope of your sweep; do not use the law to predict a 100B model from 1M-parameter points.

Hints

• Always subtract the irreducible-loss estimate before interpreting a log-log slope. The straight line is for (L - E), not for L.
• If Method 1, Method 2, and Method 3 disagree, suspect learning-rate contamination or insufficient coverage of one axis.
• Bootstrap CIs assume independent points. Runs that share seeds, data order, or eval splits violate that assumption.
• The D \approx 20N rule is a headline for a specific constant set and compute range. Your fitted law may give 15 or 35; what matters is that the number comes from your data.
• Keep the largest, longest runs in the fit. They anchor E, and E anchors everything else.

Key Terms