Downloads + tokenizes a dataset in the registry (dataset_name) and saves output .npy files to data_dir.

Main training program. This function will run the following in the order provided: 1) initialize Megatron. 2) setup model, optimi

Prints a nice summary of the top-level contents in a checkpoint dictionary.

Prints a nice summary of the top-level contents in a checkpoint dictionary.

Print arguments.

Simple GPU memory report.

Save a model checkpoint.

Init method based on N(0, sigma/sqrt(2*num_layers).

Set model parallel rank.

Set the model parallel size

Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current vocabula

Optional DeepSpeed Activation Checkpointing features. Gives access to partition activations, contiguous memory optimizations and cpu checkpoin

Fills the input Tensor with values according to the method described in Transformers without Tears: Improving the Normalization of Self-Attention

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

replaces items in val1 with items in val2 where boolean = True

Writes the docs dictionary to markdown format

Tokenize a string.

Train the model function.

Single training step with DeepSpeed's pipeline parallel engine.

Trains a tokenizer on all the json files in `input_dir` and saves it to `save_path` :param input_dir: input directory containing jsonl files

Log training information such as losses, timing, etc.

At runtime, checks types are actually the type specified.

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks

Iterator over input documents using lm_dataformat. Should be able to handle jsons / texts / other compressed formats. Also filters out empty