Class Adam

Adam is an optimization algorithm that can used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data.

Adam was presented by Diederik Kingma from OpenAI and Jimmy Ba from the University of Toronto in their 2015 ICLR paper(poster) titled “Adam: A Method for Stochastic Optimization“. I will quote liberally from their paper in this post, unless stated otherwise.

Inheritance

System.Object

BaseOptimizer

Adam

Inherited Members

BaseOptimizer.Name

BaseOptimizer.LearningRate

BaseOptimizer.Momentum

BaseOptimizer.DecayRate

Namespace: SiaNet.Optimizers

Assembly: SiaNet.dll

Syntax

public class Adam : BaseOptimizer

Constructors

Adam(Single, Single, Single, Single, Single, Boolean)

Initializes a new instance of the Adam class.

Declaration

public Adam(float lr = 0.01F, float beta_1 = 0.9F, float beta_2 = 0.999F, float decayRate = 0F, float epsilon = 1E-08F, bool amsgrad = false)

Parameters

Type	Name	Description
System.Single	lr	Initial learning rate for the optimizer
System.Single	beta_1	The beta 1 value.
System.Single	beta_2	The beta 2 value.
System.Single	decayRate	Learning rate decay over each update.
System.Single	epsilon	Fuzz factor. Lowest float value but > 0.
System.Boolean	amsgrad	Whether to apply the AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and Beyond".

Properties

AmsGrad

Whether to apply the AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and Beyond".

Declaration

public bool AmsGrad { get; set; }

Property Value

Type	Description
System.Boolean	true if [ams grad]; otherwise, false.

Beta1

Gets or sets the beta 1 value.

Declaration

public float Beta1 { get; set; }

Property Value

Type	Description
System.Single	The beta1.

Beta2

Gets or sets the beta 2 value.

Declaration

public float Beta2 { get; set; }

Property Value

Type	Description
System.Single	The beta2.

Epsilon

Fuzz factor. Lowest float value but > 0

Declaration

public float Epsilon { get; set; }

Property Value

Type	Description
System.Single	The epsilon.

See Also

BaseOptimizer