Take a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of

Query OpenAI chat model and save the results to output_path. `instances` is a list of dictionaries, each dictionary contains a key "prompt" a

Returns `t5` style ROUGE scores. See the related implementation: https://github.com/google-research/text-to-text-transfer-transformer

Returns `t5` style ROUGE scores. See the related implementation: https://github.com/google-research/text-to-text-transfer-transformer

Call the given iterables in a round-robin fashion. For example: ``roundrobin('ABC', 'D', 'EF') --> A D E B F C``

Find the package root and run the tests for the given tasks

Find the package root and run the tests for the given tasks

Returns the beams with the highest perturbed log probabilities.

Use this to save a state dict for an FSDP model and its optimizer via :module:`torch.distributed.checkpoint` functions. This should be used d

:param rank0_only: INFO and below messages will only be emitted on the rank0 process.

Whether this task supports decontamination against model training set.

Whether this task supports decontamination against model training set.

Parses something like args1=val1,arg2=val2 Into a dictionary

Splits a string on whitespaces and records the indices of each in the original string. @:return generator((word, (start_idx, end_idx)), ...)

Splits a string on whitespaces and records the indices of each in the original string. @:return generator((word, (start_idx, end_idx)), ...)

Wrap a predictor in a loop that reads from stdin and writes to stdout. The predictor implements `predict` function that takes a single string

Wrap a predictor in a loop that reads from stdin and writes to stdout. The predictor implements `predict` function that takes a single string

Taken from the official evaluation script for v1.1 of the SQuAD dataset.

Taken from the official evaluation script for v1.1 of the SQuAD dataset.

Tokenize a file of documents using the provided tokenizer; file is expected to be a gzipped JSON lines file, each containing a field named `text`.