The 1Cycle Learning Rate Policy: Accelerating Model Training
In our pervious article (day 42) , we have explained The Power of Learning Rates in Deep Learning and Why Schedules Matter, lets now focus on 1Cycle Learning Rate to explain it in more detail :
The 1Cycle Learning Rate Policy, first introduced by Leslie Smith in 2018, remains one of the most effective techniques for optimizing model training. By 2025, it continues to prove its efficiency, accelerating convergence by up to 10x compared to traditional learning rate schedules, such as constant or exponentially decaying rates. Today, both researchers and practitioners are pushing the boundaries of deep learning with this method, solidifying its role as a key component in the training of modern AI models.
How the 1Cycle Policy Works
The 1Cycle policy deviates from conventional learning rate schedules by alternating between two distinct phases:
- Phase 1: Increasing Learning Rate – The learning rate starts low and steadily rises to a peak value (η_max). This phase promotes rapid exploration of the loss landscape, avoiding sharp local minima.
- Phase 2: Decreasing Learning Rate – Once the peak is reached, the learning rate gradually decreases to a very low value, enabling the model to fine-tune its parameters and converge on smoother, more generalizable solutions.
Momentum Cycling
Additionally, the 1Cycle policy involves cycling the momentum inversely with the learning rate. When the learning rate is high, momentum is kept low, and when the learning rate is reduced, momentum increases. This combination helps maintain smooth convergence and prevents overfitting, making it especially effective for larger learning rates in the exploratory phase.
Why Use the 1Cycle Policy?
The advantages of the 1Cycle Learning Rate Policy include:
- Faster convergence: Models can achieve peak accuracy in fewer epochs, significantly reducing training time.
- Improved generalization: The cyclic learning rate avoids sharp local minima, leading to better generalization on unseen data.
- Scalability: In 2024, 1Cycle has been successfully applied to deep architectures and large batch sizes, yielding both speed and accuracy improvements.
Implementation in Python (PyTorch)
Below is a simple PyTorch implementation of the 1Cycle Learning Rate Policy:
import torch
import torch.optim as optim
from torch.optim.lr_scheduler import OneCycleLR
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
# Define your model (e.g., a simple CNN)
model = torch.nn.Sequential(
torch.nn.Conv2d(1, 32, kernel_size=3, stride=1),
torch.nn.ReLU(),
torch.nn.MaxPool2d(kernel_size=2, stride=2),
torch.nn.Flatten(),
torch.nn.Linear(32 * 13 * 13, 10)
)
# Set up data loaders
train_loader = DataLoader(
datasets.MNIST('.', download=True, transform=transforms.ToTensor()),
batch_size=64, shuffle=True
)
# Define optimizer and loss function
optimizer = optim.SGD(model.parameters(), lr=0.1)
loss_fn = torch.nn.CrossEntropyLoss()
# Define OneCycleLR scheduler
scheduler = OneCycleLR(optimizer, max_lr=0.1,
steps_per_epoch=len(train_loader), epochs=10)
# Training loop
for epoch in range(10):
for batch_idx, (data, target) in enumerate(train_loader):
optimizer.zero_grad()
output = model(data)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()
scheduler.step() # Update learning rate per batch
print(f"Epoch {epoch+1}, Loss: {loss.item()}")
This example demonstrates how to use PyTorch’s OneCycleLR scheduler to implement the 1Cycle learning rate policy. The learning rate increases for the first half of the training and decreases for the second half, in line with the 1Cycle approach.
Conclusion :
1Cycle Learning Rate Policy has found wide use across various tasks, from image classification to natural language processing (NLP). Its success in accelerating training for transformer-based models, known for their slow convergence, has been notable. As datasets grow larger and architectures more complex, 1Cycle continues to offer consistent performance improvements.
Implementations of the 1Cycle policy are available in popular deep learning frameworks. For instance, PyTorch provides the OneCycleLR scheduler, which allows users to apply this learning rate policy easily. Similarly, DeepSpeed offers support for the 1Cycle schedule, facilitating efficient large-scale model training.
In summary, the 1Cycle Learning Rate Policy remains a powerful and versatile tool in the deep learning community, enabling faster convergence and improved performance across a range of applications.
Follow us on LinkedIn