Load from a YAML file.

Encode a string into token IDs.

Decode a list of token IDs to a string.

Write a list of token IDs to the memmap file; if only a subset of the values can be written, return the rest. Args: value

Initialize a tokenizer from a pretrained tokenizer on the HuggingFace Hub. :param identifier: The identifier of a model on the Hub t

Returns whether the given path corresponds to an existing directory.

Returns whether the given path corresponds to an existing file.

Save to a YAML file.

Load a regular state dict from the file ``fname`` within ``checkpoint_dir`` using :func:`torch.load()`. This can be used during distributed t

shortcut for model.generate

Seed all rng objects.

Given a starting state and a step function, apply beam search to find the most likely target sequences. Returns a tuple of `

Get the local rank per filesystem, meaning that, regardless of the number of nodes, if all ranks share the same filesystem then `get_fs_local_rank

Initialize a tokenizer from a file. You can create those files with ``BaseTokenizer.save()``. :param filename: The name of

save quantized model and configs to local disk

Save a regular state dict to the file ``fname`` within ``checkpoint_dir`` using :func:`torch.save()`. This can be used during distributed tra

Get the total number of parameters.

Static method for applying the softmax function. Parameters ---------- logits : np.ndarray The input to

Close the memmap file and optionally upload it to the destination (in the case of a remote path).