Momentum#

Domain: ai.onnx.preview.training
Since version: 1

Compute one iteration of stochastic gradient update with momentum. This operator can conduct the optimization of multiple tensor variables.

Inputs

R (T1): The learning rate.
T (T2): Update count of “X”. It should be a scalar.
inputs (T3): It sequentially contains the current values of optimized tensors, then their gradient tensors, and finally their momentum tensors. For example, if two tensors “X_1” and “X_2” are optimized, The expected input list would be [“X_1”, “X_2”, gradient of “X_1”, gradient of “X_2”, momentum of “X_1”, momentum of “X_2”].

Outputs

outputs (T3): It sequentially contains the new values of optimized tensors and then the new values of their momentum tensors. For example, if two tensors “X_1” and “X_2” are optimized, the output list would be [new value of “X_1,” new value of “X_2” new momentum of “X_1”, new momentum of “X_2”].

Type Constraints

T1: Constrain input types to float scalars. Allowed types: tensor(double), tensor(float).
T2: Constrain input types to 64-bit integer scalars. Allowed types: tensor(int64).
T3: Constrain input types to float tensors. Allowed types: tensor(double), tensor(float).

Examples#

test_momentum

Node:
  ai.onnx.preview.training.Momentum(R, T, X, G, V) -> (X_new, V_new)
  Attributes:
    norm_coefficient = 0.0010000000474974513
    alpha = 0.949999988079071
    beta = 0.10000000149011612
    mode = "standard"

Inputs:
  R: shape=(), dtype=float32
    0.1
  T: shape=(), dtype=int64
    0
  X: shape=(2,), dtype=float32
    [1.2, 2.8]
  G: shape=(2,), dtype=float32
    [-0.94, -2.5 ]
  V: shape=(2,), dtype=float32
    [1.7, 3.6]

Outputs:
  X_new: shape=(2,), dtype=float32
    [1.13238, 2.70772]
  V_new: shape=(2,), dtype=float32
    [0.67620003, 0.9227999 ]

test_momentum_multiple

Node:
  ai.onnx.preview.training.Momentum(R, T, X1, X2, G1, G2, H1, H2) -> (X1_new, X2_new, V1_new, V2_new)
  Attributes:
    norm_coefficient = 0.0010000000474974513
    alpha = 0.949999988079071
    beta = 0.8500000238418579
    mode = "standard"

Inputs:
  R: shape=(), dtype=float32
    0.1
  T: shape=(), dtype=int64
    0
  X1: shape=(1,), dtype=float32
    [1.]
  X2: shape=(2,), dtype=float32
    [1., 2.]
  G1: shape=(1,), dtype=float32
    [-1.]
  G2: shape=(2,), dtype=float32
    [-1., -3.]
  H1: shape=(1,), dtype=float32
    [2.]
  H2: shape=(2,), dtype=float32
    [4., 1.]

Outputs:
  X1_new: shape=(1,), dtype=float32
    [0.9099]
  X2_new: shape=(2,), dtype=float32
    [0.7199, 2.2048]
  V1_new: shape=(1,), dtype=float32
    [0.90099996]
  V2_new: shape=(2,), dtype=float32
    [ 2.8009999, -2.048    ]

test_nesterov_momentum

Node:
  ai.onnx.preview.training.Momentum(R, T, X, G, V) -> (X_new, V_new)
  Attributes:
    norm_coefficient = 0.009999999776482582
    alpha = 0.949999988079071
    beta = 1.0
    mode = "nesterov"

Inputs:
  R: shape=(), dtype=float32
    0.1
  T: shape=(), dtype=int64
    0
  X: shape=(2,), dtype=float32
    [1.2, 2.8]
  G: shape=(2,), dtype=float32
    [-0.94, -2.5 ]
  V: shape=(2,), dtype=float32
    [1.7, 3.6]

Outputs:
  X_new: shape=(2,), dtype=float32
    [1.227535, 2.95714 ]
  V_new: shape=(2,), dtype=float32
    [0.68700004, 0.9479999 ]