A batch generator implements continuous batching. This class provides automatic management of prompt processing and generation batches,

BitLinear module with memory-efficient weight handling.

A state machine that uses one Aho-Corasick trie per state to efficiently track state across a generated sequence. The transitions are provide

The output of :func:`stream_generate`. Args: text (str): The next segment of decoded text. This can be an empty string. toke

A simple linear layer used to replace Attention or MLP blocks.

A wrapper that combines an HF tokenizer and a detokenizer. Accessing any attribute other than the ``detokenizer`` is forwarded to the hugging