lmflow.optim.lamb#

Classes#

Lamb

Implements Lamb algorithm.

Module Contents#

class lmflow.optim.lamb.Lamb(params, lr: float = 0.001, betas=(0.9, 0.999), eps: float = 1e-06, weight_decay: float = 0, clamp_value: float = 10, adam: bool = False, debias: bool = False)[source]#

Bases: torch.optim.optimizer.Optimizer

Implements Lamb algorithm.

It has been proposed in Large Batch Optimization for Deep Learning: Training BERT in 76 minutes https://arxiv.org/abs/1904.00962

Note:

Reference code: cybertronai/pytorch-lamb

clamp_value[source]#
adam[source]#
debias[source]#
step(closure=None)[source]#

Performs a single optimization step.

Arguments:

closure: A closure that reevaluates the model and returns the loss.