The multi-vector counterpart of :func:`embed_columns`. Returns per-column ``(batch_size, tokens, dim)`` embeddings and their boolean scoring mask

Returns the appropriate batch sampler based on the ``batch_sampler`` argument in ``self.args``. This batch sampler class supports ``_

Get the configuration dictionary. Returns: Dictionary containing the configuration parameters

Returns the appropriate multi-dataset batch sampler based on the ``multi_dataset_batch_sampler`` argument in ``self.args``. This batc

Whether any conversation in a batch carries both of the ``query``/``document`` pair roles. Anchored on produced messages so both routes into

Whether the model embeds its inputs with a StaticEmbedding, directly or behind a Router. StaticEmbedding features are an EmbeddingBag (``input_id

A utility function to load a file from a model checkpoint. The checkpoint can be either a local directory or a model id on Hugging Face.

We simply one-hot encode the sentences. If a sentence contains a keyword, the corresponding one-hot encoding is added to the sentence

Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and return

Returns a list of possible values for the SimilarityFunction enum. Returns: list: A list of possible values for the Simi

Save the model, processor, and module config to ``output_path``.

DEPRECATED, use `push_to_hub` instead. Uploads all elements of this model to a new HuggingFace Hub repository. Args:

Stack a list of ``(B, T_i, D)`` token embeddings and their ``(B, T_i)`` masks along ``dim=1``, padding each column up to the batch-wide max token

In this context, :meth:`SentenceTransformer.encode` outputs embeddings truncated at dimension ``truncate_dim``. This may be

XTR scoring for knowledge distillation. Same global top-k scoring as :func:`xtr_scores`, but returns each query's own N-way document scores `

The LambdaLoss Framework for Ranking Metric Optimization. This loss function implements the LambdaLoss framework for ranking metric optimizat

Call the appropriate processor with the correct arguments. Dispatches based on the processor type and modality: 1. **Message modalit

Shared input front door for one side of a scoring call: masks become a single tensor (a list of per-item masks is padded to the batch), and a bare

Helper: return fraction of samples that appear in properly balanced batches (not the remainder).

Create an automatic model card and store it in the specified path. Args: path (str): The path where the model card will

Create a small Dataset with a Video feature column from synthetic video files.

The trailing embedding dimension, read without materializing a padded batch.

The device multi-vector embeddings live on, or None for numpy and empty inputs.

Get the default modality configuration for the current transformer task. Returns: tuple[ModalityConfig, str]: A tuple of (modalit

Get the attributes of the processor if available. Will be removed in the future as transformers v5 becomes the minimum requirement. R

Resolve the task a column is preprocessed with. Subclasses may add positional defaults.

Ward hierarchical clustering on cosine distance for a single 2D embedding.

Whether the loaded model was configured for FlashAttention-2 (via ``attn_implementation``).

Args: model_name_or_path (str): The name or path of the pre-trained model. token (Optional[Union[bool, str]]): The t

Load a saved model, resolving the ``modules.json`` types listed in ``module_classes`` to those classes rather than importing them. Ty

Check if a string looks like a valid URL (starts with http(s) and has no spaces).

Like ``_make_feature``, but each sample has its own real token count. ``mini_batch_num_tokens`` packs by real tokens, so it only produces uneven m

Read Stanford-NLP ColBERT settings from ``artifact.metadata`` and stash them on ``self._legacy`` for the Transformer constructor + prefix-toke

Replace each non-system message content with ``filler``, keeping everything else verbatim. Keeping system messages whole lets the derivation

Pad one chunk of a side onto the scoring device, with its mask narrowed to the chunk's local width. Padding the whole batch upfront instead would

Pad a list of variable-length multi-vector tensors into a single 3D tensor with a mask. Returns the padded tensor and either the user-provided ma

Pad a list of ``(t_i, D)`` tensors back into a ``(B, max_t, D)`` tensor. Callers guarantee a non-empty list.

Per-item count of real query tokens for MeanMaxSim scoring: the mask sum when a mask is given, the item lengths for ragged lists, and the non-zero

Mark a prompt-prefix token as special so the tokenizer emits it as a single piece. Call only with the prefixes of an existing token-prepended

Render the examples table for a dataset, saving non-text values as assets when possible. Returns: A tuple of ``(rendered_table, n

Instantiate an activation function from a dotted path string, respecting trust_remote_code.

Resolve the concrete ``*ForRetrieval`` class (ColPali, ColQwen2, ...) for ``transformer_task="retrieval"``. transformers (as of v5.13) has no ``A

Save a non-text value to the assets directory, deduplicating by content hash. Args: prefix: Prepended to filenames, e.g. ``"examp

Wraps the save_pretrained method of a model to save to a subfolder. Args: _save_pretrained_fn: The original save_pretrained function

Mirror the evaluator's student MaxSim scoring so teacher scores can be built from it.

Move one side's padded embeddings and mask onto the scoring device. The mask is checked separately, since a caller can hand in a cpu mask alongsid

Warn if the config requests bidirectional attention settings not supported by the installed transformers version.

If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT official documentation: htt

Gathers a tensor from each distributed rank into a list. Always retains gradients for the local rank's tensor, and optionally retains gradien

All-gather a ``(B, T, D)`` token-embedding tensor and its ``(B, T)`` mask across ranks, padding the token axis to the cross-rank max ``T`` first.

Determines whether the model should be exported to the backend, or if it can be loaded directly. Also update the `file_name` and `subfolder`

Warns the user to save the model if they just exported it. Args: model_name_or_path: The model name or path is_local: Whethe

Generalized function to calculate the cross-entropy loss without caching gradients.

Calculate the all-pairs InfoNCE loss, chunked over ``mini_batch_size`` rows of the score matrix. The labels are unused: the positive pair is t

Concatenate ``(B_i, T_i, D)`` token-embedding chunks and ``(B_i, T_i)`` mask chunks along ``dim=0``, padding each chunk up to the chunk-wide max t

Checks if a package is available from the correct owner.

Inspect a distillation loss's teacher target once, on its first forward. Takes the target the loss itself computed rather than recomputing a soft

Pairwise ColBERT (MaxSim) scoring for matched ``(query_i, document_i)`` pairs. Takes ``(batch_size, q_tokens, dim)`` query embeddings and ``(batc

Compute the Triplet loss from embeddings. Args: embeddings: List of embeddings Returns: Loss value