Package com.jml.optimizers

Class Adam

java.lang.Object
com.jml.optimizers.Optimizer
com.jml.optimizers.Adam
public class Adam extends Optimizer
The Adam (Adaptive Moment Estimation) optimizer. Adam is a first order gradient method for efficient stochastic optimization.

This is recommended as the default optimizer on most problems. Applies the following update rule: 
      gt = grad(wt-1)
      mt = b1*mt-1 + (1-b1)*gt
      vt = b2*vt-1 + (1-b2)*gt2

      m̂t = mt/(1-b1t)
      v̂t = vt/(1-b2t)

      wt = wt-1 - a*m̂t/(√(v̂t) + ϵ)

      Where a is the learning rate, b1 and b2 are the moment parameters,
      and ϵ is a small value to avoid division bny zero.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static String
    OPTIM_NAME
     

    Fields inherited from class com.jml.optimizers.Optimizer

    name, schedule

  • Constructor Summary

    Constructors
    Constructor
    Description
    Adam​(double learningRate, double beta1, double beta2)
    Creates a Adam optimizer with specified learning rate and parameters.

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    getDetails()
    Gets the details of this optimizer.
    linalg.Matrix[]
    step​(boolean increaseTime, linalg.Matrix... params)
    Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w.
    linalg.Matrix[]
    step​(linalg.Matrix... params)
    Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w.

    Methods inherited from class com.jml.optimizers.Optimizer

    getLearningRate, setScheduler

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

    • Adam

      public Adam​(double learningRate, double beta1, double beta2)
      Creates a Adam optimizer with specified learning rate and parameters.
      Parameters:
      learningRate - Learning rate for the Adam optimizer.
      beta1 - Exponential decay rate for first moment estimate. Must be in [0, 1)
      beta2 - Exponential decay rate for second moment estimate. Must be in [0, 1)

  • Method Details

    • step

      public linalg.Matrix[] step​(linalg.Matrix... params)
      Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w. Note, this will increase the time step of the Adam optimizer. To not increase the time step see step(boolean, Matrix[]).
      Specified by:
      step in class Optimizer
      Parameters:
      params - An array of matrices strictly containing {w, wGrad, v, m} where w is the matrix containing the weights to apply the update to, wGrad is the gradient of the objective function with respect to w, v is the second moment estimate, and v is the first moment estimate.
      Returns:
      The result of applying the update rule of the optimizer to the matrix w.

    • step

      public linalg.Matrix[] step​(boolean increaseTime, linalg.Matrix... params)
      Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w.
      Specified by:
      step in class Optimizer
      Parameters:
      increaseTime - Flag for increasing the timestamp of the Adam optimizer.
      params - An array of matrices strictly containing {w, wGrad, v, m} where w is the matrix containing the weights to apply the update to, wGrad is the gradient of the objective function with respect to w, v is the second moment estimate, and v is the first moment estimate.
      Returns:
      The result of applying the update rule of the optimizer to the matrix w.

    • getDetails

      public String getDetails()
      Gets the details of this optimizer.
      Specified by:
      getDetails in class Optimizer
      Returns:
      Important details of this optimizer as a string.