Optimizer

An optimizer is used to optimize model parameters.

Algorithems Provided by PyTorch

Adadelta	Implements Adadelta algorithm.	Notes
Adagrad	Implements Adagrad algorithm.
Adam	Implements Adam algorithm.	Adam
AdamW	Implements AdamW algorithm.
SparseAdam	Implements lazy version of Adam algorithm suitable for sparse tensors.
Adamax	Implements Adamax algorithm (a variant of Adam based on infinity norm).
ASGD	Implements Averaged Stochastic Gradient Descent.
LBFGS	Implements L-BFGS algorithm, heavily inspired by minFunc.
NAdam	Implements NAdam algorithm.
RAdam	Implements RAdam algorithm.
RMSprop	Implements RMSprop algorithm.
Rprop	Implements the resilient backpropagation algorithm.
SGD	Implements stochastic gradient descent (optionally with momentum).	SGD

torch.optim Base Class implements an abstract method step that updates the parameters.

Scheduler

Docs

torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs.

Reference

• torch.optim