Computes scaled dot product attention on query, key and value tensors, using an optional attention mask if passed, and applying dropo

Split a full s3 path into the bucket name and path.

Splits hidden dim into attn_head_size and num_attention_heads

Annoyingly, torchmetrics only supports tensors as input, not raw values. So we have to convert raw values into tensors. From

Annoyingly, torchmetrics only supports tensors as input, not raw values. So we have to convert raw values into tensors. From

This unsharding implementation consists of: 1. Loading each shard on a separate process and copying their sharded tensors to shared

Further, torchmetrics can't handle single-instance calls when given tensors. It always needs the first dimension of the tensors to be

Further, torchmetrics can't handle single-instance calls when given tensors. It always needs the first dimension of the tensors to be

Expand tensors in a state dictionary from `(batch_size, *)` to `(batch_size * beam_size, *)`.

Add special tokens in-place (if not already present) to the given token IDs.

Returns `t5` style BLEU scores. See the related implementation: https://github.com/google-research/text-to-text-transfer-transformer/

Returns `t5` style BLEU scores. See the related implementation: https://github.com/google-research/text-to-text-transfer-transformer/

From catwalk. https://github.com/allenai/catwalk/blob/main/catwalk/model.py

Calculates the perplexity of the language model. Parameters ---------- n_ctx : int The context size.