Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you

Derives additional configuration values necessary for training from the current config

Ensure fixed arguments for a model are the same for the input arguments and the one retrieved from checkpoint.

Compile helper function at runtime. Make sure this is invoked on a single process.

Converts a sequence of tokens into ids using the vocab.

take in a loaded yaml from NeoX and assign relevant values to HF config. Returns: GPTNeoXConfig() object

From pretrain_gpt2:forward_step()

iterates through data and handles wrap around

Set the groups to none.

downloads dataset

enable Tee logs based on the configured logdir

Build filename's path if it does not already exists.

Evaluation. neox_args: NeoX Arguments forward_step_fn: function with args `neox_args, timers, data_iterator & model that w

Checks if the dataset is present

Expands an `attention_config` list in the following format: [ [['attention_type_1', ..., `attention_type_n`], 12] ]

Filters the logits using top_k / top_p, filling any filtered vocab items with filter_value (defaults to -inf). This function has been mostly

Forward step.

Generates samples from an input file and writes them to an output file. Reads prompts from neox_args.sample_input_file and writes completion

Generates samples unconditionially (no prompt) and yields them in a dictionary. neox_args: NeoXArgs. model: a Megatron model maximu

Generates samples unconditionially (no prompt) and yields them in a dictionary. neox_args: NeoXArgs. model: a Megatron model number

Get triangular attention mask for a given sequence length / device.

Generate a batch

Gets the git commit hash of your current repo (if it exists)

Build the learning rate scheduler.

Learning rate decay functions from: https://openreview.net/pdf?id=BJYwwY9ll pg. 4

Build the model.

Set up the optimizer.

Divide params into with-weight-decay and without-weight-decay groups. Layernorms and biases will have no weight decay but the rest will.

Get dataset splits from comma or '/' separated string list.

Init method based on N(0, sigma).

inserts the layers in `layers` into the pipe model at `idx`.

Read list of objects from a JSON lines file.

Generate text/sample model

Get dict of `resource_pool[hostname] = [list of GPU ranks]` using hostfile, include and exclude args. Modified from: `deepspeed.launcher.runn

pads context lengths in context_tokens with pad_id to equal neox_args.seq_length, and returns the padded batch and the new lengths. cont

s -> (s0,s1), (s1,s2), (s2, s3), ...

LM logits using word embedding weights.