Use ftfy (https://github.com/LuminosoInsight/python-ftfy) to fix text encodings

A modification of get_batch() to work with the latest batch instead of an iterator.

Return my rank for the data parallel group.

Calculate the global rank corresponding to a local rank zero in the data parallel group.

Return world size for the data parallel group.

Get the IO parallel group the caller rank belongs to.

Provide extra arguments required for merging.

Return my rank for the pipe parallel group.

Return world size for the pipe parallel group.

Greedy until is lm_eval harness' way to say "do greedy generation" - necessary for some tasks. the eval harness dispatches requests t

Sets up the model for inference by turning on k/v caching (if specified) and setting `parallel output` of the final layer to false, s

Initialize model data parallel groups. Arguments: model_parallel_size: number of GPUs used to parallelize model. Let's say we h

Dictionary from vocab id token to text token.

True if bitsandbytes optimizers are available

Useful for code segments that may be accessed with or without mpu initialization

returns variables within megatron args

Merge file for tokenizer

Check if model and data parallel groups are initialized.

name of dataset

Number of documents in the dataset (if known)

Number of workers to use in preprocessing

returns the datatype on the basis of configured precision

Entry point for @distributed_test().

Save the tokenizer vocabulary and merge files to a directory.

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

verify consume_deepy_args processes command line arguments without config dir

Verify consume_deepy_args processes command line arguments without yaml suffix. Also test the hostfile CLI arg

verify consume_deepy_args processes command line arguments without yaml suffix