.. _op_ai_onnx_preview_training_Momentum:

Momentum
========

- **Domain**: ``ai.onnx.preview.training``
- **Since version**: 1

Compute one iteration of stochastic gradient update with momentum.
This operator can conduct the optimization of multiple tensor variables.

**Inputs**

- **R** (*T1*): The learning rate.
- **T** (*T2*): Update count of "X". It should be a scalar.
- **inputs** (*T3*): It sequentially contains the current values of optimized tensors, then their gradient tensors, and finally their momentum tensors. For example, if two tensors "X_1" and "X_2" are optimized, The expected input list would be ["X_1", "X_2", gradient of "X_1", gradient of "X_2", momentum of "X_1", momentum of "X_2"].

**Outputs**

- **outputs** (*T3*): It sequentially contains the new values of optimized tensors and then the new values of their momentum tensors. For example, if two tensors "X_1" and "X_2" are optimized, the output list would be [new value of "X_1," new value of "X_2" new momentum of "X_1", new momentum of "X_2"].

**Type Constraints**

- **T1**: Constrain input types to float scalars.
  Allowed types: tensor(double), tensor(float).
- **T2**: Constrain input types to 64-bit integer scalars.
  Allowed types: tensor(int64).
- **T3**: Constrain input types to float tensors.
  Allowed types: tensor(double), tensor(float).

Examples
--------

**test_momentum**

.. code-block:: text

    Node:
      ai.onnx.preview.training.Momentum(R, T, X, G, V) -> (X_new, V_new)
      Attributes:
        norm_coefficient = 0.0010000000474974513
        alpha = 0.949999988079071
        beta = 0.10000000149011612
        mode = "standard"

.. code-block:: text

    Inputs:
      R: shape=(), dtype=float32
        0.1
      T: shape=(), dtype=int64
        0
      X: shape=(2,), dtype=float32
        [1.2, 2.8]
      G: shape=(2,), dtype=float32
        [-0.94, -2.5 ]
      V: shape=(2,), dtype=float32
        [1.7, 3.6]

    Outputs:
      X_new: shape=(2,), dtype=float32
        [1.13238, 2.70772]
      V_new: shape=(2,), dtype=float32
        [0.67620003, 0.9227999 ]

**test_momentum_multiple**

.. code-block:: text

    Node:
      ai.onnx.preview.training.Momentum(R, T, X1, X2, G1, G2, H1, H2) -> (X1_new, X2_new, V1_new, V2_new)
      Attributes:
        norm_coefficient = 0.0010000000474974513
        alpha = 0.949999988079071
        beta = 0.8500000238418579
        mode = "standard"

.. code-block:: text

    Inputs:
      R: shape=(), dtype=float32
        0.1
      T: shape=(), dtype=int64
        0
      X1: shape=(1,), dtype=float32
        [1.]
      X2: shape=(2,), dtype=float32
        [1., 2.]
      G1: shape=(1,), dtype=float32
        [-1.]
      G2: shape=(2,), dtype=float32
        [-1., -3.]
      H1: shape=(1,), dtype=float32
        [2.]
      H2: shape=(2,), dtype=float32
        [4., 1.]

    Outputs:
      X1_new: shape=(1,), dtype=float32
        [0.9099]
      X2_new: shape=(2,), dtype=float32
        [0.7199, 2.2048]
      V1_new: shape=(1,), dtype=float32
        [0.90099996]
      V2_new: shape=(2,), dtype=float32
        [ 2.8009999, -2.048    ]

**test_nesterov_momentum**

.. code-block:: text

    Node:
      ai.onnx.preview.training.Momentum(R, T, X, G, V) -> (X_new, V_new)
      Attributes:
        norm_coefficient = 0.009999999776482582
        alpha = 0.949999988079071
        beta = 1.0
        mode = "nesterov"

.. code-block:: text

    Inputs:
      R: shape=(), dtype=float32
        0.1
      T: shape=(), dtype=int64
        0
      X: shape=(2,), dtype=float32
        [1.2, 2.8]
      G: shape=(2,), dtype=float32
        [-0.94, -2.5 ]
      V: shape=(2,), dtype=float32
        [1.7, 3.6]

    Outputs:
      X_new: shape=(2,), dtype=float32
        [1.227535, 2.95714 ]
      V_new: shape=(2,), dtype=float32
        [0.68700004, 0.9479999 ]