Load existing results from file if it exists

Load existing results from file if it exists.

Load existing results from file

Load the MATH-500 dataset. Returns: list[dict]: The dataset of problems.

Load the OptiLLM Bench dataset.

Load a source dataset with error handling

Initializes context, query, tokenizer, logging, and config from an input string. Args: initial_query (str): Input string containing

Main evaluation function.

Main entry point

Main execution function

Run all tests.

Main test function

Make n attempts to solve a problem and return all responses and predictions. Args: problem (str): The problem text model

Map requested model to provider-specific name

Merge strategies that are very similar to each other.

Direct proxy approach that passes through all parameters to the underlying endpoint. Args: client: OpenAI client instance mo

Helper function to normalize algebraic expressions.

Helper function to normalize fractions.

Helper function to normalize intervals.

Helper function to normalize matrices and vectors.

Ensure all message content fields are strings, not lists. Some models don't handle list-format content correctly.

Basic normalization for comparing responses. Removes extra whitespace, punctuation at ends, and lowercases.

Compare if two numeric strings represent the same value.

Parse command line arguments

Parse the model string to extract base model and adapter IDs

Parse the selection response to extract the chosen candidate number and reasoning. Args: response: The LLM's comparison response

Perform targeted searches based on identified gaps in the current draft Prioritizes HIGH priority gaps (placeholder tags) first

Perform Round-Trip Correctness evaluation. Args: query: Original query model: Model name to use Returns:

Prepare globals dictionary for Z3/SymPy code execution. WARNING: This is NOT a security sandbox. The name "execution_globals" reflects t

Main online processing with warmup and early termination. Args: messages: Input messages Re

Process a single sample from the dataset.

Process a single sample from the dataset.

Prune strategies with poor performance.

Push the dataset to HuggingFace Hub

Quantize model to 4-bit precision using bitsandbytes

Query OptILLM with the specified approach

Rank responses based on correctness and token efficiency.

Rank the responses using the LLM.

Rates completions for the Best of N step of CePO. Each completion is rated on a scale of 1 to 10 individually. Parameters: system_pr

Rates completions for the Best of N step of CePO. Completions are rated pairwise against each other in both orders (A vs B and B vs A). Para

Generate response with ThinkDeeper's controlled thinking process using MLX

Generate response with ThinkDeeper's controlled thinking process

Record an attempt to use this strategy.

Refine a strategy based on its application to a specific problem. Args: strategy: The strategy to refine problem: The pr

Refine a strategy based on new insights.

Release a slot for this provider.

Force reload configuration from disk.

Remove and close a session.

Remove steering hooks from a model. Args: hooks: List of (hook, handle) tuples

Remove a <think>...</think> block from the response text, if present. Args: response (str): Raw model output. Returns:

Remove <think>...</think> blocks from the response.

TTD-DR (Test-Time Diffusion Deep Researcher) main algorithm Implements the core diffusion process with: 1. Preliminary draft

Run T iterations of RSA-style aggregation Args: workspace: MARS workspace containing solutions request_id: R

Run the complete evaluation

Run all IMO25 MARS tests

Main plugin function that implements system prompt learning. By default, the plugin runs in inference-only mode, which uses existing str

Run all MARS tests

Run the warmup phase to generate initial traces and calibrate threshold. Args: messages: Input messages

Prepare code for safe execution by removing problematic visualization code.

Save raw response data (including logprobs) to a separate file.

Save a single result to the results file.

Save a single result to the results file with incremental updates

Save a single result to the results file.

Save a single result with incremental updates

Save a single result to the results file.

Save evaluation results to files.

Save evaluation results to files

Select a provider from the list