Shift input ids one token to the right.

从表为 【seq_len，seq_len】 的 span_labels 提取 只需要计算 loss 的 token 的labels，提取之后，size ->【token_len, token_len】

Runs basic whitespace cleaning and splitting on a piece of text.

Runs basic whitespace cleaning and splitting on a piece of text.

Runs basic whitespace cleaning and splitting on a piece of text.

Do not add eos again if user already added it.

convert into overlapping chunks. Chunk size = 2w, overlap size = w

Computes global attention mask by putting attention on all tokens before `sep_token_id` if `before_sep_token is True` else after `sep_token_i

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

force one of token_ids to be generated by setting prob of all other tokens to 0

Initialize the weights

Checks whether `char` is a punctuation character.

Strips accents from a piece of text.

Recursively sets an use_cache to `value` on a list of pytorch modules, if they have a use_cache attribute. use_cache is used to decide whethe

Same as _sliding_chunks_query_key_matmul but for attn_probs and value tensors. Returned tensor will be of the same shape as `attn_pro

Matrix multiplication of query and key tensors using with a sliding window attention pattern. This implementation splits the input in

A fast approximation of the cumulative distribution function of the standard normal.

Aggregate a batch data. batch = [ins1_dict, ins2_dict, ..., insN_dict] batch_data = {'sentence':[ins1_sentence, ins2_sentence

This function is copy of `random_spans_helper <https://github.com/google-research/ text-to-text-transfer-transformer/blob/84f8bcc14b5f2c03de51bd35

Compute what the p_mean_variance output would have been, should the model's score function be conditioned by cond_fn. See co

Compute what the p_mean_variance output would have been, should the model's score function be conditioned by cond_fn. See co

If comm is provided, average all numerical stats across that comm

Convert the torso of the model to float16.

A counter for the `thop` package to count the operations in an attention operation. Meant to be used like: macs, params = thop.pr

Sentinel ids creation given the indices that should be masked. The start indices of each mask are replaced by the sentinel ids in inc

Merge segments A and B, add [CLS] and [SEP] and build tokentypes.

Defaults for image and classifier training.

Ensure that numerator is divisible by the denominator and return the division value.

Ensure that numerator is divisible by the denominator and return the division value.

repetition penalty (from CTRL paper https://arxiv.org/abs/1909.05858).

Ensure that numerator is divisible by the denominator.

Puts sentinel mask on `input_ids` and fuse consecutive mask tokens into a single mask token by deleting. This will reduce the sequenc

取出对应的text，然后拼接起来