Gelu#

Domain: ai.onnx
Since version: 20

Gelu takes one input data (Tensor ) and produces one output data (Tensor ) where the gaussian error linear units function, $y = 0.5 * x * (1 + erf(x/sqrt(2)))$ is applied to the tensor elementwise. If the attribute “approximate” is set to “tanh”, the function estimation, $y = 0.5 * x * (1 + Tanh(sqrt(2/pi) * (x + 0.044715 * x^3)))$ is used and applied to the tensor elementwise.

Inputs

X (T): Input tensor

Outputs

Y (T): Output tensor

Attributes

approximate (string): Gelu approximation algorithm: "tanh", "none"``(default).”none”: do not use approximation.”tanh”``: use tanh approximation.

Type Constraints

T: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

Examples#

test_cc_gelu_default_1

Node:
  Gelu(X) -> (Y)

Inputs:
  X: shape=(3,), dtype=float32
    [-1.,  0.,  1.]

Outputs:
  Y: shape=(3,), dtype=float32
    [-0.15865526,  0.        ,  0.8413447 ]

test_cc_gelu_default_2

Node:
  Gelu(X) -> (Y)
  Attributes:
    approximate = "none"

Inputs:
  X: shape=(2, 3), dtype=float32
    [[-2. , -1. , -0.5],
     [ 0.5,  1. ,  2. ]]

Outputs:
  Y: shape=(2, 3), dtype=float32
    [[-0.04550028, -0.15865526, -0.15426877],
     [ 0.34573123,  0.8413447 ,  1.9544997 ]]

test_cc_gelu_default_bfloat16

Node:
  Gelu(X) -> (Y)

Inputs:
  X: shape=(4,), dtype=bfloat16
    [-1, 0, 1, 0.5]

Outputs:
  Y: shape=(4,), dtype=bfloat16
    [-0.158203, 0, 0.839844, 0.345703]

test_cc_gelu_default_float16

Node:
  Gelu(X) -> (Y)

Inputs:
  X: shape=(3,), dtype=float16
    [-1.,  0.,  1.]

Outputs:
  Y: shape=(3,), dtype=float16
    [-0.1587,  0.    ,  0.8413]

test_cc_gelu_tanh_1

Node:
  Gelu(X) -> (Y)
  Attributes:
    approximate = "tanh"

Inputs:
  X: shape=(3,), dtype=float32
    [-1.,  0.,  1.]

Outputs:
  Y: shape=(3,), dtype=float32
    [-0.158808,  0.      ,  0.841192]

test_cc_gelu_tanh_2

Node:
  Gelu(X) -> (Y)
  Attributes:
    approximate = "tanh"

Inputs:
  X: shape=(2, 3), dtype=float32
    [[-2. , -1. , -0.5],
     [ 0.5,  1. ,  2. ]]

Outputs:
  Y: shape=(2, 3), dtype=float32
    [[-0.04540229, -0.158808  , -0.154286  ],
     [ 0.345714  ,  0.841192  ,  1.9545977 ]]

test_cc_gelu_tanh_float16

Node:
  Gelu(X) -> (Y)
  Attributes:
    approximate = "tanh"

Inputs:
  X: shape=(2, 3), dtype=float16
    [[-2. , -1. , -0.5],
     [ 0.5,  1. ,  2. ]]

Outputs:
  Y: shape=(2, 3), dtype=float16
    [[-0.0454, -0.1588, -0.1543],
     [ 0.3457,  0.8413,  1.955 ]]