after initialization of default or loaded values a number of functions are performed in order to calculate values, assert con

Sample index mapping is a 2D array with sizes [number-of-samples + 1, 2] where [..., 0] contains the index into `doc_idx` and [..., 1] is the

Set autoresume start time.

Just a wrapper to massage inputs/outputs from pipeline.

In this method, the model doesn't do any generation, but just returns log likelihoods for the next token, which eval harness uses to

Rescale parameters to convert SP initialization to μP initialization. Warning: This method is NOT idempotent and should be called only once

Add token-type embedding. This function is provided so we can add token-type embeddings in case the pretrained model does not have it.

returns variables of all args

base data directory

Broadcast data from rank zero of each model parallel group to the members of the same model parallel group. Arguments: keys: list of

Converts a sequence of ids in BPE tokens using the vocab.

Distributed Debugger that will insert a py debugger on rank `rank` and pause all other distributed processes until debugging is complete.

returns a dict containing variables within deepspeed config

returns variables within deepspeed runner

Initialize torch.distributed and execute the user function.

Second-level decorator for dist_test. This actually wraps the function.

A decorator for executing a function (e.g., a unit test) in a distributed manner. This decorator manages the spawning and joining of processes, in

Helper function to be used with checkpoint() Adapted from torch.utils.checkpoint:checkpoint_sequential()