lmflow.optim.adabound

Classes

AdaBound

Implement the AdaBound optimization algorithm.

Module Contents

class lmflow.optim.adabound.AdaBound(params, lr: float = 0.001, betas=(0.9, 0.999), final_lr: float = 0.1, gamma: float = 0.001, eps: float = 1e-08, weight_decay: float = 0, amsbound: bool = False)

Bases: torch.optim.optimizer.Optimizer

Implement the AdaBound optimization algorithm.

See “Adaptive Gradient Methods with Dynamic Bound of Learning Rate”: https://arxiv.org/abs/1902.09843.

Reference implementation: Luolc/AdaBound.

base_lrs

__setstate__(state) → None

step(closure=None)

Performs a single optimization step.

Parameters:

closure – A closure that reevaluates the model and returns the loss.