include_text_kernels.h#

namespace onnx_light

Alias that makes onnx-light headers compatible with code that references ONNX_LIGHT_NAMESPACE (the macro used in the standard onnx package).

Set to ONNX_LIGHT_NAMESPACE so both names resolve to the same namespace.

Symbol-visibility attribute for the public onnx-light C++ API.

Defined as empty because onnx-light does not require explicit __declspec(dllexport) or __attribute__((visibility("default"))) annotations — visibility is controlled at the shared-library level. The macro is provided so that vendored ONNX headers that decorate their declarations with ONNX_API compile without modification.

Namespace alias so that ONNX C++ code (and consumers such as onnxruntime) that refers to the literal onnx namespace — rather than the ONNX_NAMESPACE macro — resolves to the onnx-light namespace. The standard onnx package lives in namespace onnx; onnx-light uses onnx_light (via ONNX_LIGHT_NAMESPACE), so this alias keeps onnx-light a true drop-in. It is only introduced when the onnx-light namespace differs from onnx.

namespace onnx_kernels

namespace kernel

class RegexFullMatch : public onnx_light::onnx_kernels::kernel::KernelBase #

#include <include_text_kernels.h>

Reference implementation of the RegexFullMatch operator (ai.onnx, since opset 20).

Performs an element-wise full-match regex test on a tensor(string) input and produces a tensor(bool) output with the same shape. The regex pattern is provided either as the pattern attribute on the underlying node or directly to operator() as a std::string.

The ONNX specification references the RE2 <>_ regex syntax. This reference kernel does not link against RE2; it uses the C++ standard library’s ECMAScript regex grammar (std::regex_match), which is sufficient for the common constructs exercised by the upstream backend tests (anchors, character classes, alternation, quantifiers, and groups). Patterns that rely on RE2-specific syntax not supported by std::regex will throw std::invalid_argument.

Public Functions

Tensor operator()(const Tensor &x, const std::string &pattern) const#: Allocating overload. Returns a freshly allocated tensor(bool) with the same shape as x. Each output byte is 1 when the corresponding input string is fully matched by pattern and 0 otherwise.

void operator()(const Tensor &x, const std::string &pattern, Tensor &output) const#: In-place overload. output must already be a tensor(bool) with the same shape as x and a pre-sized data buffer of x.element_count() bytes.

inline explicit KernelBase(const KernelContext &ctx)#

Public Static Functions

static inline constexpr bool CanRunInPlace() noexcept#: Output bytes depend on the input contents and on the regex pattern; the output buffer cannot safely alias the input string_data buffer (different layout / element size).

class StringConcat : public onnx_light::onnx_kernels::kernel::KernelBase #

#include <include_text_kernels.h>

Element-wise concatenation of two tensor(string) inputs with NumPy-style bidirectional broadcasting.

Public Functions

Tensor operator()(const Tensor &x, const Tensor &y) const#

void operator()(const Tensor &x, const Tensor &y, Tensor &output) const#

inline explicit KernelBase(const KernelContext &ctx)#

Public Static Functions

static inline constexpr bool CanRunInPlace() noexcept#: Output bytes depend on both inputs, so the output buffer cannot safely alias either input buffer.

class StringNormalizer : public onnx_light::onnx_kernels::kernel::KernelBase #

#include <include_text_kernels.h>

Reference implementation of the StringNormalizer operator (ai.onnx, since opset 10). Removes elements found in stopwords from the input string tensor (case-sensitively or not) and applies the requested case_change_action ("LOWER", "UPPER" or "NONE") to the surviving elements.

Accepts only [C]-shaped or [1, C]-shaped tensor(string) inputs. When every element is dropped, the output is a single empty string with shape [1] (for [C] input) or [1, 1] (for [1, C] input).

Case folding and comparison use the "C" locale (ASCII): the locale attribute, when present, is ignored. This matches the behavior of the upstream onnx reference implementation, which only supports the "en_US"-equivalent ASCII semantics.

Public Types

enum class CaseChangeAction#

case_change_action attribute values.

Values:

enumerator kNone#

enumerator kLower#

enumerator kUpper#

Public Functions

Tensor operator()(const Tensor &x, CaseChangeAction case_change_action = CaseChangeAction::kNone, bool is_case_sensitive = false, const std::vector<std::string> &stopwords = {}) const#: Allocating overload. stopwords may be empty.

void operator()(const Tensor &x, CaseChangeAction case_change_action, bool is_case_sensitive, const std::vector<std::string> &stopwords, Tensor &output) const#: In-place overload. output must be a tensor(string) with the shape returned by :cpp:func:ComputeOutputShape and with a matching number of pre-allocated entries in string_data.

inline explicit KernelBase(const KernelContext &ctx)#

Public Static Functions

static CaseChangeAction ParseCaseChangeAction(const std::string &value)#: Converts the string form of the case_change_action attribute ("NONE", "LOWER" or "UPPER") into the matching enumerator. Throws std::invalid_argument for any other value.

static std::vector<int64_t> ComputeOutputShape(const std::vector<int64_t> &input_shape, int64_t kept)#: Computes the output shape given the input shape and the number of surviving (non-stopword) elements. Encapsulates the [C] → [max(1, kept)] / [1, C] → [1, max(1, kept)] rule.

static inline constexpr bool CanRunInPlace() noexcept#: Output bytes depend on the input contents and on the stopwords set; the output buffer cannot safely alias the input buffer.

class StringSplit : public onnx_light::onnx_kernels::kernel::KernelBase #

#include <include_text_kernels.h>

Reference implementation of the StringSplit operator (ai.onnx, since opset 20).

Splits each element of the input tensor(string) according to the delimiter attribute and returns:

Y — a padded string tensor whose shape is input.shape + [M], where M is the maximum number of substrings produced by any input element;
Z — an INT64 tensor of shape input.shape storing the number of substrings produced for each input element.

When delimiter is empty, the operator follows the ONNX reference semantics and splits on consecutive whitespace.

Public Functions

std::pair<Tensor, Tensor> operator()(const Tensor &x, const std::string &delimiter = "", int64_t maxsplit = -1) const#

inline explicit KernelBase(const KernelContext &ctx)#

Public Static Functions

static inline constexpr bool CanRunInPlace() noexcept#

class TfIdfVectorizer : public onnx_light::onnx_kernels::kernel::KernelBase #

#include <include_text_kernels.h>

Reference implementation of the TfIdfVectorizer operator (ai.onnx, since opset 9).

Extracts n-grams from a [C]- or [N, C]-shaped integer (tensor(int32) or tensor(int64)) or string (tensor(string)) input and produces a tensor(float) count ("TF") / weight ("IDF" / "TFIDF") vector with last dimension max(ngram_indexes) + 1.

The kernel parameters mirror the operator attributes:

mode — one of "TF", "IDF" or "TFIDF".
min_gram_length / max_gram_length — inclusive range of n-gram sizes to extract.
max_skip_count — maximum number of tokens to skip between consecutive elements of an n-gram (skip_distance ranges from 1 to max_skip_count + 1).
ngram_counts — starting offsets of 1-grams, 2-grams, … in the pool (CSR-style).
ngram_indexes — output coordinate of each pool entry (parallel to pool_*).
pool_int64s or pool_strings — exactly one must be set; the type of the active pool must match the input dtype.
weights — optional per-output weight; defaults to all ones when IDF/TFIDF is selected and weights is empty.

Public Types

enum class Mode#

Weighting criteria. Matches the mode attribute of the TfIdfVectorizer operator.

Values:

enumerator kTF#

enumerator kIDF#

enumerator kTFIDF#

Public Functions

Tensor operator()(const Tensor &x, Mode mode, int64_t min_gram_length, int64_t max_gram_length, int64_t max_skip_count, const std::vector<int64_t> &ngram_counts, const std::vector<int64_t> &ngram_indexes, const std::vector<int64_t> &pool_int64s, const std::vector<std::string> &pool_strings, const std::vector<float> &weights) const#: Allocating overload. pool_int64s must be non-empty when x is an integer tensor; pool_strings must be non-empty when x is a string tensor. weights may be empty (treated as all-ones for IDF / TFIDF).

void operator()(const Tensor &x, Mode mode, int64_t min_gram_length, int64_t max_gram_length, int64_t max_skip_count, const std::vector<int64_t> &ngram_counts, const std::vector<int64_t> &ngram_indexes, const std::vector<int64_t> &pool_int64s, const std::vector<std::string> &pool_strings, const std::vector<float> &weights, Tensor &output) const#: In-place overload. output must already be a tensor(float) with the shape returned by :cpp:func:ComputeOutputShape and a pre-sized data buffer.

inline explicit KernelBase(const KernelContext &ctx)#

Public Static Functions

static Mode ParseMode(const std::string &value)#: Parses the string form of the mode attribute ("TF", "IDF" or "TFIDF"). Throws std::invalid_argument for any other value.

static std::vector<int64_t> ComputeOutputShape(const std::vector<int64_t> &input_shape, int64_t output_size)#: Computes the output shape of a TfIdfVectorizer node given the input shape and the size of the ngram_indexes attribute. Input rank must be 1 or 2.

static inline constexpr bool CanRunInPlace() noexcept#: Output values depend on every input element and on the entire pool; the output buffer cannot safely alias the input buffer (different dtype / shape).