Build the size on rank 0 and broadcast.

Check that all the keys have the same target data type.

Performs invalid character removal and whitespace cleanup on text.

Performs invalid character removal and whitespace cleanup on text.

Performs invalid character removal and whitespace cleanup on text.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

num_splits: whether to parallelize over the seqlen_k dimension (num_splits > 1) or not (num_splits = 1). num_splits=0 means it will be set by

num_splits: how much to parallelize over the seqlen_q dimension. num_splits=0 means it will be set by an internal heuristic. We're exposing n

compute global attn indices required throughout forward pass

Computes the index of the first occurrence of `sep_token_id`.

Get slopes for Alibi positional embedding n : int = number of heads. For best performance, restrict n to a power of 2.

Checks whether CP is the codepoint of a CJK character.

Checks whether CP is the codepoint of a CJK character.

Checks whether CP is the codepoint of a CJK character.

Checks whether `char` is a control character.

Checks whether `chars` is a control character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a punctuation character.

Checks whether `char` is a whitespace character.

Checks whether `chars` is a whitespace character.

Checks whether `chars` is a whitespace character.

Make causal mask used for bi-directional self-attention.

shift every row 1 step right, converting columns into diagonals. Example:: chunked_hidden_states: [ 0.4983, 2.6918,

pads rows and then flips rows and columns

A helper function to pad tokens and mask to work with implementation of Longformer self-attention.

Get the prior KL term for the variational lower-bound, measured in bits-per-dim. This term can't be optimized, as it only de

read file return format : [ ['EU', 'B-ORG'], ['rejects', 'O'], ['German', 'B-MISC'], ['call', 'O'], ['to', 'O'], ['boycott',

read file return format : [ ['EU', 'B-ORG'], ['rejects', 'O'], ['German', 'B-MISC'], ['call', 'O'], ['to', 'O'], ['boycott',

Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transf

Splits punctuation on a piece of text.

Splits punctuation on a piece of text.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Strips accents from a piece of text.

类似 -3 -2 -1 0 1 2 -3 -2 -1 0 1 2 -3 -2 -1 0 1 2 转换为 0 1 2 -1 0 1 -2 -1 0

Adds whitespace around any CJK character.

Adds whitespace around any CJK character.

Adds whitespace around any CJK character.

Truncates a sequence pair in place to the maximum length.

Truncates a sequence pair in place to the maximum length.