StringNormalizer

• Domain: ai.onnx

• Since version: 10

StringNormalization performs string operations for basic cleaning. This operator has only one input (denoted by X) and only one output (denoted by Y). This operator first examines the elements in the X, and removes elements specified in “stopwords” attribute. After removing stop words, the intermediate result can be further lowercased, uppercased, or just returned depending the “case_change_action” attribute. This operator only accepts [C]- and [1, C]-tensor. If all elements in X are dropped, the output will be the empty value of string tensor with shape [1] if input shape is [C] and shape [1, 1] if input shape is [1, C].

Inputs

• X (tensor(string)): UTF-8 strings to normalize

Outputs

• Y (tensor(string)): UTF-8 Normalized strings

Examples

test_cc_string_normalizer_all_dropped

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    stopwords = ['a', 'b']

Inputs:
  x: shape=(2,), dtype=object
    ['a', 'b']

Outputs:
  y: shape=(1,), dtype=object
    ['']

test_cc_string_normalizer_case_sensitive

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    is_case_sensitive = 1
    stopwords = ['the']

Inputs:
  x: shape=(3,), dtype=object
    ['The', 'the', 'cat']

Outputs:
  y: shape=(2,), dtype=object
    ['The', 'cat']

test_cc_string_normalizer_case_sensitive_lower

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    case_change_action = "LOWER"
    is_case_sensitive = 1
    stopwords = ['monday']

Inputs:
  x: shape=(4,), dtype=object
    ['monday', 'tuesday', 'wednesday', 'thursday']

Outputs:
  y: shape=(3,), dtype=object
    ['tuesday', 'wednesday', 'thursday']

test_cc_string_normalizer_lower

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    case_change_action = "LOWER"

Inputs:
  x: shape=(3,), dtype=object
    ['Hello', 'World', 'FOO']

Outputs:
  y: shape=(3,), dtype=object
    ['hello', 'world', 'foo']

test_cc_string_normalizer_nostopwords_nochangecase

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    is_case_sensitive = 1

Inputs:
  x: shape=(2,), dtype=object
    ['monday', 'tuesday']

Outputs:
  y: shape=(2,), dtype=object
    ['monday', 'tuesday']

test_cc_string_normalizer_upper

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    case_change_action = "UPPER"
    is_case_sensitive = 0
    stopwords = ['a']

Inputs:
  x: shape=(1, 4), dtype=object
    [['A', 'hello', 'a', 'world']]

Outputs:
  y: shape=(1, 2), dtype=object
    [['HELLO', 'WORLD']]

test_cc_strnormalizer_export_monday_casesensintive_lower

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    case_change_action = "LOWER"
    is_case_sensitive = 1
    stopwords = ['monday']

Inputs:
  x: shape=(4,), dtype=object
    ['monday', 'tuesday', 'wednesday', 'thursday']

Outputs:
  y: shape=(3,), dtype=object
    ['tuesday', 'wednesday', 'thursday']

test_cc_strnormalizer_export_monday_casesensintive_nochangecase

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    is_case_sensitive = 1
    stopwords = ['monday']

Inputs:
  x: shape=(4,), dtype=object
    ['monday', 'tuesday', 'wednesday', 'thursday']

Outputs:
  y: shape=(3,), dtype=object
    ['tuesday', 'wednesday', 'thursday']

test_cc_strnormalizer_export_monday_casesensintive_upper

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    case_change_action = "UPPER"
    is_case_sensitive = 1
    stopwords = ['monday']

Inputs:
  x: shape=(4,), dtype=object
    ['monday', 'tuesday', 'wednesday', 'thursday']

Outputs:
  y: shape=(3,), dtype=object
    ['TUESDAY', 'WEDNESDAY', 'THURSDAY']

test_cc_strnormalizer_export_monday_empty_output

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    case_change_action = "UPPER"
    is_case_sensitive = 1
    stopwords = ['monday']

Inputs:
  x: shape=(2,), dtype=object
    ['monday', 'monday']

Outputs:
  y: shape=(1,), dtype=object
    ['']

test_cc_strnormalizer_export_monday_insensintive_upper_twodim

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    case_change_action = "UPPER"
    stopwords = ['monday']

Inputs:
  x: shape=(1, 6), dtype=object
    [['Monday', 'tuesday', 'wednesday', 'Monday', 'tuesday', 'wednesday']]

Outputs:
  y: shape=(1, 4), dtype=object
    [['TUESDAY', 'WEDNESDAY', 'TUESDAY', 'WEDNESDAY']]

test_cc_strnormalizer_nostopwords_nochangecase

Node:
  StringNormalizer(x) -> (y)
  Attributes:
    is_case_sensitive = 1

Inputs:
  x: shape=(2,), dtype=object
    ['monday', 'tuesday']

Outputs:
  y: shape=(2,), dtype=object
    ['monday', 'tuesday']