Load a huggingface tokenizer and try to infer the type of streaming detokenizer to use. Note, to use a fast streaming tokenizer, pass a local

Load any fine-tuned adapters / layers. Args: model (nn.Module): The neural network model. adapter_path (str): Path to the ad

Compute log-likelihood of generating a continuation from a context. Downstream tasks should attempt to use loglikelihood instead of other