Learning Rate Scheduler Visualizer

Configure a schedule, see the LR curve update live, then copy the PyTorch code.

πŸ”’100% Client-Side.Β Everything runs in your browser β€” no data is sent to any server.
PyTorch Code
import torch import torch.nn as nn model = nn.Linear(10, 1) # replace with your model from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR optimizer = torch.optim.Adam(model.parameters(), lr=0.001) warmup = LinearLR( optimizer, start_factor=1e-8, end_factor=1.0, total_iters=10, ) cosine = CosineAnnealingLR( optimizer, T_max=90, eta_min=0.000001, ) scheduler = SequentialLR( optimizer, schedulers=[warmup, cosine], milestones=[10], ) for epoch in range(100): train(...) scheduler.step()

What is a learning rate scheduler?

The learning rate controls how large a step the optimizer takes each iteration. Starting with a high LR speeds up early training, but can cause the loss to oscillate or diverge near a minimum. A learning rate scheduler automatically adjusts the LR during training so you can enjoy fast early convergence and precise fine-tuning near the end.

Without scheduling, practitioners typically choose a fixed LR that is a compromise β€” not aggressive enough to converge fast, not small enough to settle well. Schedulers remove this trade-off.

Schedule formulas and PyTorch examples

Cosine Annealing

Smoothly decays from lr_max to lr_min following a cosine curve. No sudden drops β€” the optimizer slows gradually, which works well for image classification and NLP fine-tuning.

# Formula: lr = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(Ο€ * t / T)) scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100, eta_min=1e-6)
Step Decay

Multiplies the LR by gamma every step_size epochs. Simple and predictable. Common in ResNet training where LR is dropped by Γ—0.1 at epochs 30, 60, and 90.

# lr = base_lr * gamma^(floor(epoch / step_size)) scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
Linear Warmup + Cosine Decay

Ramps LR linearly from ~0 to base_lr over warmup epochs, then applies cosine decay. The warmup stabilizes early training when model weights are random β€” especially important for Transformers.

from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR warmup = LinearLR(optimizer, start_factor=1e-8, end_factor=1.0, total_iters=10) cosine = CosineAnnealingLR(optimizer, T_max=90, eta_min=1e-6) scheduler = SequentialLR(optimizer, schedulers=[warmup, cosine], milestones=[10])
Exponential Decay

Multiplies LR by gamma every epoch. Aggressive β€” with gamma=0.95 and 100 epochs the LR falls to ~0.6% of its starting value. Use a gamma close to 1 (e.g. 0.99) for gentle decay.

# lr_t = base_lr * gamma^t scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)
ReduceLROnPlateau

Monitors a metric (usually validation loss). If it does not improve for patience epochs, the LR is multiplied by factor. Adaptive and robust β€” great when you do not know in advance when the loss will plateau.

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau( optimizer, mode='min', # reduce when metric stops decreasing factor=0.5, # new_lr = lr * factor patience=5, # epochs to wait before reducing min_lr=1e-6, ) for epoch in range(num_epochs): val_loss = validate(model, val_loader) scheduler.step(val_loss) # pass the monitored metric

How to choose a scheduler

  • Training from scratch (vision): Step decay at Γ—0.1 every 30–40 epochs is the classic choice for ResNet-style models.
  • Transformers / fine-tuning: Linear warmup + cosine decay. Warmup epochs typically 5–10% of total training steps.
  • Unknown training dynamics: ReduceLROnPlateau is safest β€” it adapts to your actual loss curve.
  • Quick experiments: Cosine annealing with no warmup. Simple, smooth, works well out of the box.
  • Rule of thumb for base LR: Use a learning rate finder (e.g. PyTorch Lightning's trainer.tune()) before picking a schedule.

Related Tools