Retrieves a single item from the dataset with the option to only return a portion of the item. get(idx) is the same as [idx] but get(

Clones the sampler chain and its internal state.

Evaluate the batch of tokens using the transformer model. This method executes the forward pass. If the KV cache is heavily fragment

Detokenize a list of tokens. Args: tokens: The list of tokens to detokenize. prev_tokens: The list of previous tokens

Updates the cache with the new `state` for the layer `layer_idx`. Parameters: state (`Tuple[torch.Tensor]`):

Map the message roles.

Explicitly free the model from memory.

Clears all stored checkpoints and resets memory tracking.

Converts a cache in the legacy cache format into an equivalent `RecurrentCache`.

Get the first system message.

Generate text from a prompt. Args: prompt: The prompt to generate text from. suffix: A suffix to append to the generated text. If

初始化 Worker，返回一个异步任务，当全部 worker 都加载成功后完成 Args: worker_num: Worker 数量 model_config: 模型配置 batch_siz

Tokenize a string. Args: text: The utf-8 encoded string to tokenize. add_bos: Whether to add a beginning of sequence

Create a Llama model from a pretrained model name or path. This method requires the huggingface_hub package. You can install it with `

Safely aborts any ongoing text generation. Useful for async API environments or UI interruption buttons.

r""" Args: r (torch.Tensor): reception of shape `(B, H, T, K)`. Alias: q, query in linear attention. k (torch.Tensor):

Format the prompt with the no-colon-single style.

Manually free LlamaModel and Vocab/Lora resources.