Adagrad#

Domain: ai.onnx.preview.training
Since version: 1

Compute one iteration of ADAGRAD, a stochastic gradient based optimization algorithm. This operator can conduct the optimization of multiple tensor variables.

Inputs

R (T1): The initial learning rate.
T (T2): The update count of “X”. It should be a scalar.
inputs (T3): The current values of optimized tensors, followed by their respective gradients, followed by their respective accumulated squared gradients.For example, if two tensor “X_1” and “X_2” are optimized, The input list would be [“X_1”, “X_2”, gradient of “X_1”, gradient of “X_2”, accumulated squared gradient of “X_1”, accumulated squared gradient of “X_2”].

Outputs

outputs (T3): Updated values of optimized tensors, followed by their updated values of accumulated squared gradients. For example, if two tensor “X_1” and “X_2” are optimized, the output list would be [new value of “X_1,” new value of “X_2” new accumulated squared gradient of “X_1”, new accumulated squared gradient of “X_2”].

Type Constraints

T1: Constrain input types to float scalars. Allowed types: tensor(double), tensor(float).
T2: Constrain input types to 64-bit integer scalars. Allowed types: tensor(int64).
T3: Constrain input and output types to float tensors. Allowed types: tensor(double), tensor(float).

Examples#

test_adagrad

Node:
  ai.onnx.preview.training.Adagrad(R, T, X, G, H) -> (X_new, H_new)
  Attributes:
    norm_coefficient = 0.0010000000474974513
    epsilon = 9.999999747378752e-06
    decay_factor = 0.10000000149011612

Inputs:
  R: shape=(), dtype=float32
    0.1
  T: shape=(), dtype=int64
    0
  X: shape=(1,), dtype=float32
    [1.]
  G: shape=(1,), dtype=float32
    [-1.]
  H: shape=(1,), dtype=float32
    [2.]

Outputs:
  X_new: shape=(1,), dtype=float32
    [1.0576962]
  H_new: shape=(1,), dtype=float32
    [2.998001]

test_adagrad_multiple

Node:
  ai.onnx.preview.training.Adagrad(R, T, X1, X2, G1, G2, H1, H2) -> (X1_new, X2_new, H1_new, H2_new)
  Attributes:
    norm_coefficient = 0.0010000000474974513
    epsilon = 9.999999747378752e-06
    decay_factor = 0.10000000149011612

Inputs:
  R: shape=(), dtype=float32
    0.1
  T: shape=(), dtype=int64
    0
  X1: shape=(1,), dtype=float32
    [1.]
  X2: shape=(2,), dtype=float32
    [1., 2.]
  G1: shape=(1,), dtype=float32
    [-1.]
  G2: shape=(2,), dtype=float32
    [-1., -3.]
  H1: shape=(1,), dtype=float32
    [2.]
  H2: shape=(2,), dtype=float32
    [4., 1.]

Outputs:
  X1_new: shape=(1,), dtype=float32
    [1.0576962]
  X2_new: shape=(2,), dtype=float32
    [1.0446854, 2.0948617]
  H1_new: shape=(1,), dtype=float32
    [2.998001]
  H2_new: shape=(2,), dtype=float32
    [4.998001, 9.988004]