BatchNormalization - 6 vs 9#

Next section compares an older to a newer version of the same operator after both definition are converted into markdown text. Green means an addition to the newer version, red means a deletion. Anything else is unchanged.

Files changed (1) hide show
  1. BatchNormalization6 → BatchNormalization9 +31 -21
BatchNormalization6 → BatchNormalization9 RENAMED
@@ -1 +1 @@
1
1
  Carries out batch normalization as described in the paper
2
2
  https://arxiv.org/abs/1502.03167. Depending on the mode it is being run,
3
3
  there are multiple cases for the number of outputs, which we list below:
4
4
  Output case #1: Y, mean, var, saved_mean, saved_var (training mode)
5
5
  Output case #2: Y (test mode)
6
- For previous (depreciated) non-spatial cases, implementors are suggested
7
- to flatten the input shape to (N x C*D1*D2 ..*Dn) before a BatchNormalization Op.
8
- This operator has **optional** inputs/outputs. See ONNX <https://github.com/onnx/onnx/blob/master/docs/IR.md>_ for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument's name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.
9
-
10
6
  **Attributes**
11
7
  * **epsilon**:
12
- The epsilon value to use to avoid division by zero.
8
+ The epsilon value to use to avoid division by zero, default is
9
+ 1e-5f.
10
+ * **is_test**:
11
+ If set to nonzero, run spatial batch normalization in test mode,
12
+ default is 0.
13
13
  * **momentum**:
14
14
  Factor used in computing the running mean and variance.e.g.,
15
- running_mean = running_mean * momentum + mean * (1 - momentum).
15
+ running_mean = running_mean * momentum + mean * (1 - momentum),
16
+ default is 0.9f.
17
+ * **spatial**:
18
+ If true, compute the mean and variance across all spatial elements
19
+ If false, compute the mean and variance across per feature.Default
20
+ is 1.
16
21
  **Inputs**
17
22
  * **X** (heterogeneous) - **T**:
18
- Input data tensor from the previous operator; dimensions are in the
23
+ Input data tensor from the previous operator; dimensions for image
24
+ case are (N x C x H x W), where N is the batch size, C is the number
25
+ of channels, and H and W are the height and the width of the data.
26
+ For non image case, the dimensions are in the form of (N x C x D1 x
27
+ D2 ... Dn), where N is the batch size.
19
- form of (N x C x D1 x D2 ... Dn), where N is the batch size, C is
20
- the number of channels. Statistics are computed for every channel of
21
- C over N and D1 to Dn dimensions. For image data, input dimensions
22
- become (N x C x H x W). The op also accepts single dimension input
23
- of size N in which case C is assumed to be 1
24
28
  * **scale** (heterogeneous) - **T**:
25
- Scale tensor of shape (C).
29
+ The scale as a 1-dimensional tensor of size C to be applied to the
30
+ output.
26
31
  * **B** (heterogeneous) - **T**:
27
- Bias tensor of shape (C).
32
+ The bias as a 1-dimensional tensor of size C to be applied to the
33
+ output.
28
34
  * **mean** (heterogeneous) - **T**:
29
- running (training) or estimated (testing) mean tensor of shape (C).
35
+ The running mean (training) or the estimated mean (testing) as a
36
+ 1-dimensional tensor of size C.
30
37
  * **var** (heterogeneous) - **T**:
31
- running (training) or estimated (testing) variance tensor of shape
38
+ The running variance (training) or the estimated variance (testing)
32
- (C).
39
+ as a 1-dimensional tensor of size C.
33
40
  **Outputs**
34
41
  Between 1 and 5 outputs.
35
42
  * **Y** (heterogeneous) - **T**:
36
- The output tensor of the same shape as X
43
+ The output tensor of the same shape as X.
37
44
  * **mean** (optional, heterogeneous) - **T**:
38
- The running mean after the BatchNormalization operator.
45
+ The running mean after the BatchNormalization operator. Must be in-
46
+ place with the input mean. Should not be used for testing.
39
47
  * **var** (optional, heterogeneous) - **T**:
40
- The running variance after the BatchNormalization operator.
48
+ The running variance after the BatchNormalization operator. Must be
49
+ in-place with the input var. Should not be used for testing.
41
50
  * **saved_mean** (optional, heterogeneous) - **T**:
42
51
  Saved mean used during training to speed up gradient computation.
52
+ Should not be used for testing.
43
53
  * **saved_var** (optional, heterogeneous) - **T**:
44
54
  Saved variance used during training to speed up gradient
45
- computation.
55
+ computation. Should not be used for testing.
46
56
  **Type Constraints**
47
57
  * **T** in (
48
58
  tensor(double),
49
59
  tensor(float),
50
60
  tensor(float16)
51
61
  ):
52
62
  Constrain input and output types to float tensors.