Initialize affine weight for model parallel on GPU.

Reads a jsonl file.

Reads a jsonl file.

Transpose a 3D tensor [b, s, np*hn] into a 4D tensor with size [b, np, s, hn].

Transpose a 3D tensor [b, s, np*hn] into a 4D tensor with size [b, np, s, hn].

Evaluate a function without caching intermediate activations, allowing for reduced memory at the expense of extra compute in the backward pas

Extracts a class from a module file, present in the local folder or repository of a model. <Tip warning={true}> Calling this function w

Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (

Converts a config key to the corresponding module.

Prune a linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_grad=True.

Zero out the parameters of a module and return it.

Converts an index (integer) in a token (str) using the vocab.

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Gather tensors and concatinate along the last dimension.

All-reduce the the input tensor across model parallel group.

Split the tensor along its last dimension and keep the corresponding slice.

Get a term for the variational lower-bound. The resulting units are bits (rather than nats, as one might expect). This allow

Build relative position according to the query and key We assume the absolute position of query \\(P_q\\) is range from (0, query_size) and

Returns: Tensor1, Tensor2 Tensor1: the tensor latent z with shape [batch, nsamples, nz]

Creates the predictions for the masked LM objective. Note: Tokens here are vocab ids and not text tokens.

retrieves the activation function specified in config

Get the data parallel group the caller rank belongs to.

See base class.

Gets a collection of `InputExample`s for the train set.

See base class.

Gets a collection of `InputExample`s for the train set.