Applies quantization to the model weights. Args: model (nn.Module): The model to be quantized. config (dict): Model configur

Set the self.linear layer and recompute self.m.

Return the weight of linear layer and dequantize it if is quantized

Make a new SequenceStateMachine or fetch it if we 've made it before. Return also a dictionary that maps the token sequences in the state

Check if model's cache can be trimmed.

Ops-based reference implementation for prompt prefill (sequential loop). Supports both scalar and vectorized gating. Shapes: - q,

Generate a single response packet based on response type (stream or not), completion type and parameters. Args:

Update all instances of type(layer) to use gradient checkpointing.

Load and initialize the model from a given path. Args: model_path (Path): The path to load the model from. lazy (bool): If F

Get the next batch of responses. Returns: Tuple of prompt processing responses and generation responses.

Set up and return the argument parser.

Uploads the model to Hugging Face hub. Args: path (str): Local path to the model. upload_repo (str): Name of the HF repo to

Build an Aho-Corasick trie from the provided sequences See https://en.wikipedia.org/wiki/Aho–Corasick_algorithm .

Extract name from quoted string.

Returns info dicts for the top `top_n` tokens from `logprobs`

Parse a single call:name{args} regex match into a tool call dict.

Perform a single generation step. Returns: Tuple of token list and logprobs list.

Rearrange the cache into temporal order.

Tokenize a request and split the prompt into segments. Returns a tuple * prompt - Full list of tokens * segments - A lis

Apply RoPE with blocked layout. Args: x: Input tensor in (B, H, T, D) format offset: Position offset for KV caching base

Dequantize the quantized layers in the model. Args: model (nn.Module): The model with quantized layers. Returns: nn.Mod