Use this to load a state dict for an FSDP model and its optimizer via :module:`torch.distributed.checkpoint` functions. This should be used d

Load model state from a distributed FSDP model checkpoint created from :func:`save_fsdp_model_and_optim_state()`. Note that ``model`` should

Compute full log-likelihood of a string, with no truncation, for perplexity computation - We will use the full max context length of the model

Compute full log-likelihood of a string, with no truncation, for perplexity computation - We will use the full max context length of the model

Finds the longest common prefix among a list of strings.

Run quantization.

Recursively list all files in the source directories and create a corresponding list of destination.

Parse a git remote URL into a GitHub (account, repo) pair. :raises InvalidRemoteError: If the URL can't be parsed correctly.

Append doc_ids to each example so that they are processed together in the metric

Take a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of