Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.

Gets the list of labels for this data set.

Calculate the global rank corresponding to a local rank zero in the model parallel group.

Get the pipe parallel group the caller rank belongs to.

Get a list that maps a sample index to a starting sentence index, end sentence index, and length

Loads the tokenizer configuration from a pretrained model tokenizer configuration. Args: pretrained_model_name_or_path (`str` or `os

Check if the current word piece is the starting piece (BERT).

Check if the current word piece is the starting piece (BERT).

Copy model parameters into a (differently-shaped) list of full-precision parameters.

Defaults for image training.

Compute the KL divergence between two gaussians. Shapes are automatically broadcasted, so batches can be compared to scalars, among othe

Fills the input Tensor with a (semi) orthogonal matrix, as described in Exact solutions to the nonlinear dynamics of learning in deep linear neura

sample from posterior Gaussian family Args: mu: Tensor Mean of gaussian distribution with shape (batch, nz)

Rotates half the hidden dims of the input.

_summary_ Args: model (_type_): _description_ context_tokens_tensor (Tensor): [bs, seq_len] context_length_tensor (Tensor

Create sinusoidal timestep embeddings. :param timesteps: a 1-D Tensor of N indices, one per batch element. These may b

Serializes this instance to a Python dictionary. Override the default [`~PretrainedConfig.to_dict`]. Returns: `Dict[str,

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (vocabulary

Truncates a pair of sequences to a maximum sequence length.

LlamaRMSNorm is equivalent to T5LayerNorm