Experiment Tracking and Hyperparameter Sweeps

The Problem

Three weeks into a project every team has a directory that looks like this:

textCopyrun_47_final.pt
run_48_final.pt
run_49_backup_final.pt
best_model_so_far.pt
best_model_so_far_v2.pt

The names are lies. The numbers do not encode anything. And the file that actually matters — the one that reached 94.2% validation accuracy — is already impossible to recreate, because nobody wrote down the learning rate, the random seed, or whether that run used dropout 0.3 or 0.5.

This is the ad-hoc tracking death spiral:

• Memory fails first. You tweak lr, momentum, and batch_size on Monday. By Wednesday you cannot match a plot to the code that produced it.
• Spreadsheets and CSV logs fail second. Columns drift between runs, filenames are manually typed, and the row with the best val_loss has no commit hash, no seed, and no hyperparameter snapshot.
• Git annotations fail third. Committing after every run pollutes history and still does not capture intermediate metrics, package versions, or the exact random initialization.

The real cost is not lost files; it is lost decisions. You ship a model because it "looked better last week," but you cannot prove it, reproduce it, or explain why it beats the alternative. Experiment tracking turns those decisions into data.

The Concept

What an experiment tracker actually tracks

A tracker is just a structured log with five pieces:

Production tools such as Weights & Biases, MLflow, and TensorBoard wrap this same model. They add a web UI, shared project pages, report builders, and (in W&B's case) an offline mode that writes to a local directory and syncs later. The concepts are identical whether you are using a managed service or a 200-line stdlib implementation.

Sweeps: grid, random, and Bayesian

A sweep is a loop that tries many configs and records every result. The three classic strategies differ in how they pick the next config.

Grid search evaluates every combination of specified values. It is exhaustive and easy to audit, but its cost grows exponentially with the number of hyperparameters.

Random search samples configs from distributions. Bergstra & Bengio (2012) showed that random search usually finds better configurations than grid search with the same budget, because most problems have low effective dimensionality: only 1-2 hyperparameters really matter. Grid search wastes most evaluations on the unimportant dimensions, while random search covers the important ones more densely.

Bayesian optimization learns from past evaluations. It fits a cheap surrogate model (often a Gaussian process) to the (config, score) pairs it has seen, then uses an acquisition function such as Expected Improvement to pick the next point:

where f^* is the best score observed so far. Bayesian optimization typically reaches a good configuration with 2-5x fewer evaluations than random search, at the cost of more bookkeeping.

ASHA: stop bad runs early

Not every configuration deserves the full training budget. If a model is still terrible after a few epochs, it will almost certainly still be terrible after hundreds. Successive Halving (and its async variant ASHA) allocates small budgets to many runs, keeps the best fraction, and gives the survivors more epochs.

The rung schedule with multiplier \eta looks like this:

At rung \eta^k r, each surviving run has been trained for \eta^k r steps. Runs are ranked by the target metric and only the top 1/\eta are promoted. The median stopping rule is the same idea in disguise: kill any run whose intermediate result is worse than the median of completed runs at the same step.

ASHA is particularly brutal in the best way. A configuration that would have wasted 27 epochs is often stopped after 1 or 3, freeing compute for the promising ones.

Reading a parallel coordinates plot

Parallel coordinates are the sweep equivalent of a spreadsheet. Each vertical axis is a hyperparameter or a metric; each line is one run. To read one:

• Look for clusters of good lines — configs that repeatedly land in the same region.
• Look for axes with little spread near the top — those hyperparameters do not matter much.
• Look for sharp transitions — small changes in one axis that separate good runs from bad ones. That is the knob you should tune next.

Build It

The lesson ships a reusable stdlib tracker in code/main.py. No network calls, no daemon, no account. Import it into any training script and you get the same primitives as a production tool.

Tracker: one JSONL record per run

pythonCopyfrom main import Tracker

tracker = Tracker(".runs")
metrics = [
    {"step": 1, "loss": 2.3, "accuracy": 0.21},
    {"step": 2, "loss": 1.8, "accuracy": 0.45},
]
run_id = tracker.log_run({"lr": 0.01, "batch_size": 64, "seed": 42}, metrics)

Each .jsonl file contains one line with the run id, timestamp, environment snapshot, config, and full metrics history. The format is append-friendly, diff-friendly, and trivial to load with json.loads.

Sweeper: grid, random, and ASHA

pythonCopyfrom main import Sweeper, toy_objective

tracker = Tracker(".runs")
sweeper = Sweeper(tracker, toy_objective, max_steps=27, rung_multiplier=3)

param_grid = {
    "lr": [0.05, 0.1, 0.2],
    "momentum": [0.0, 0.5, 0.9],
}
configs = sweeper.grid_configs(param_grid)
survivors = sweeper.run_asha(configs, metric="loss", minimize=True)

run_asha trains every config at the first rung (1 step), keeps the top third, trains those at 3 steps, keeps the top third again, and so on until the full budget of 27 steps. Stopped runs are still logged, so you can study exactly what got killed and why.

The toy task

The built-in objective is a 2-D ill-conditioned quadratic:

optimized with SGD + momentum. It is convex, deterministic, and fast, which makes it perfect for demonstrating that a sweep actually finds the best (lr, momentum) pair and that two runs with the same seed produce bit-identical histories.

Queries on the run database

pythonCopybest = tracker.best_run("loss", minimize=True)
fast = tracker.filter_runs(lambda r: r["config"].get("lr") == 0.1)

best_run returns the run whose final metric is best; filter_runs returns every run whose config satisfies a predicate. These two functions replace the "grep through log files" workflow.

Run the demo

consoleCopypython3 main.py

The demo sweeps nine (lr, momentum) combinations, prints a leaderboard, reports how many runs ASHA stopped early, and verifies reproducibility by logging the same seeded run twice and checking that the metric histories match exactly.

Verify It

code/tests/test_main.py locks down the behaviors that make a tracker useful:

• the environment snapshot contains Python version and platform;
• log_run writes a JSONL file that load_runs reads back exactly;
• best_run correctly selects minimum and maximum metrics;
• filter_runs selects runs by config predicate;
• grid and random config generators produce the expected number of combinations and respect distribution bounds;
• ASHA stops unpromising runs early while still promoting survivors;
• identical seeded configs produce identical metric histories.

Why This Matters for Engineers

• Shared run records. A tracker is the single source of truth for a team. Nobody has to ask "what was the learning rate for run_48?" because the answer is in the run record.
• Reproducibility. A config plus a seed plus an environment snapshot is a reproducibility bundle. Two months later you can recreate the exact training curve.
• Cost control. ASHA and median pruning can cut a sweep's compute bill by 50-80% by killing runs that are clearly doomed. On a cloud GPU budget that is real money.
• Decision evidence. The leaderboard and parallel-coordinates view turn "I think this model is better" into "this config achieves lower validation loss with 40% fewer epochs."
• Offline-first safety. Production trackers support offline logging for a reason: experiments run on air-gapped clusters, spot instances, and planes. Writing JSONL locally first and syncing later is the robust pattern.

Hints

• Always log the random seed in the config. A great result you cannot reproduce is a ghost.
• If the best hyperparameter value is at the edge of your search space, widen the space. The optimum is probably outside it.
• A sweep is not a replacement for understanding. Before you run 100 trials, run 10 and plot learning curves to make sure the budget is large enough for the metric to be meaningful.
• Parallel coordinates reward large sample sizes. With fewer than 20 runs, the plot is mostly noise.
• When comparing two runs, compare whole curves, not just final numbers. A run that converges slower may generalize better.

Key Terms