Converts an index (integer) in a token (str) using the vocab.

Converts an index (integer) in a token (str) using the vocab.

Converts a token (str) in an id using the vocab.

Converts a token (str) in an id using the vocab.

Initialize the weights

Tokenize a string.

Returns a tokenized string. We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Converts a sequence of tokens (string) in a single string.

Converts a sequence of tokens (string) in a single string.

Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids

Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (

r""" Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing