Package com.jml.optimizers

Class Momentum

java.lang.Object
com.jml.optimizers.Optimizer
com.jml.optimizers.Momentum
public class Momentum extends Optimizer
The momentum based gradient descent optimizer. This is similar to vanilla gradient descent but will dampen oscillations during optimization by helping accelerate the optimization in the relevant direction where the gradient may be steeper than in other directions.

Applies the following update rule: 
      vt = y*vt-1 + a*grad( wt )
      wt+1 = wt - vt

      Where a is the learning rate and y is the momentum with 0 <= y <= 1
Note: If the momentum term is zero, then this optimizer is equivalent to Vanilla Gradient Descent.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static String
    OPTIM_NAME
     

    Fields inherited from class com.jml.optimizers.Optimizer

    name, schedule

  • Constructor Summary

    Constructors
    Constructor
    Description
    Momentum​(double learningRate)
    Creates a Momentum optimizer with specified learning rate.
    Momentum​(double learningRate, double momentum)
    Creates a Momentum optimizer with specified learning rate and momentum.

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    getDetails()
    Gets the details of this optimizer.
    linalg.Matrix[]
    step​(boolean flag, linalg.Matrix... params)
    Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w.
    linalg.Matrix[]
    step​(linalg.Matrix... params)
    Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w.

    Methods inherited from class com.jml.optimizers.Optimizer

    getLearningRate, setScheduler

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

    • Momentum

      public Momentum​(double learningRate)
      Creates a Momentum optimizer with specified learning rate. A default momentum of 0.9 will be used. To specify a momentum, see Momentum(double, double).
      Parameters:
      learningRate - The learning rate to use in the update rule of the Momentum optimizer.

    • Momentum

      public Momentum​(double learningRate, double momentum)
      Creates a Momentum optimizer with specified learning rate and momentum.
      Parameters:
      learningRate - Learning rate to use when applying this optimizer.
      momentum - Momentum value to use when applying this optimizer.

  • Method Details

    • step

      public linalg.Matrix[] step​(linalg.Matrix... params)
      Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w.
      Specified by:
      step in class Optimizer
      Parameters:
      params - An array of matrices containing strictly {w, wGrad, v} where w is a matrix containing the weights to apply the update to, wGrad is the gradient of the objective function with respect to w, and v is the update vector for the momentum optimizer.
      Returns:
      The result of applying the update rule of the optimizer to the matrix w.

    • step

      public linalg.Matrix[] step​(boolean flag, linalg.Matrix... params)
      Steps the optimizer a single iteration by applying the update rule of the optimizer to the matrix w.
      Specified by:
      step in class Optimizer
      Parameters:
      flag - Does nothing for the momentum optimizer.
      params - An array of matrices containing strictly {w, wGrad, v} where w is a matrix containing the weights to apply the update to, wGrad is the gradient of the objective function with respect to w, and v is the update vector for the momentum optimizer.
      Returns:
      The result of applying the update rule of the optimizer to the matrix w.

    • getDetails

      public String getDetails()
      Gets the details of this optimizer.
      Specified by:
      getDetails in class Optimizer
      Returns:
      Important details of this optimizer as a string.