Log a provider API call and response. Args: request_id: The request ID for this conversation provider_reques

Perform Google search and return results

Load and cache configuration. Args: path: Optional path to config file force_reload: Force reload ev

Start logging a new conversation. Args: client_request: The original request from the client approach: The o

Returns the token length of a prompt using the given tokenizer.

Finalize and save the conversation to disk. Args: request_id: The request ID for this conversation

Main MARS function implementing multi-agent reasoning with parallel execution Args: system_prompt: System-level instructions

Save strategies and metrics to disk.

Set up test environment with local inference

Get new log entries since test started

Add a request to be batched Args: request_data: The request data dictionary Returns:

Close the browser driver

Get all solutions that have passed verification

Remove reasoning tags from model responses for clean final output. Removes common reasoning tags like: - <think></think> - <thin

Helper function to normalize number representation.

Detect if CAPTCHA is present on the page

Call LLM with fallback on reasoning effort levels. This function wraps `llm_call` with retry and degradation logic to handle two main cl

Log an error for this conversation. Args: request_id: The request ID for this conversation error: Error mess

Log the final response sent back to the client. Args: request_id: The request ID for this conversation final

Start health checking in background

Validate a provider response and return its message content. Guards against the empty / None / length-truncated responses that would

Get context window size: try provider /models endpoint first, then config fallback.

Extract thinking content from <think>...</think> tags and the response after. Args: response: The model's response Retu

Get OpenAI client configured for local optillm

Load configuration from file

Log MCP communication in detail

Set the batch processing function Args: processor_func: Function that takes list of request data and returns lis

Shutdown the batcher and all background threads

Wait for CAPTCHA to be resolved with user confirmation

Generate a small chunk of text using MLX with proper sampler

Mock completions.create with configurable delay

Execute a tool on an MCP server This function creates a fresh connection for each tool execution to ensure reliability.

Create a Strategy instance from a dictionary.

Generate completions with optional logprobs

Get the captured log output

Get existing searcher or create a new one

Helper function that sends a prompt to the chat-based LLM API and returns the generated response along with usage logging. Args: cli

Get batching statistics

Get a strategy by its ID.

Normalize the answer string for comparison.

Remove <think>...</think> blocks from the response. If there's a </think> tag, only keep the content after it.

Convert a tagged conversation string or list of strings into a list of messages. If the input doesn't contain User:/Assistant: tags, return it as

Compute a signature for semantic similarity matching

Simple heuristic classification when model isn't available. Args: text: The query text Retu

Get reasoning effort level based on agent temperature

Add a new agent solution to the workspace

Analyze thinking patterns in the response. Extract tokens between <think> and </think> tags and count thought transitions. Args:

Calculate final evaluation metrics

Runs `gen_function` concurrently over a list of context chunks. Args: gen_function (Callable): Function to call with each chunk and

Connect to the server and discover capabilities using appropriate transport

Connect to server using SSE transport and discover capabilities

Connect to server using stdio transport and discover capabilities

Count tokens within <think>...</think> tags in the given text. Args: text: The text to analyze tokenizer: Optional tokenizer

Create completion with load balancing, failover, and timeout handling

Evaluate if the response matches the ground truth based on category. Args: response: Model's response ground_truth: Corr

Execute the pipeline n times and return n responses. Args: n (int): Number of times to run the pipeline approaches (list): L

Execute tool using stdio transport

Execute a tool using an existing session

Extract URLs from search results and fetch their content using readurls plugin Returns content and list of sources with metadata

Extract the numerical answer from a math solution response using unified extraction. AIME problems expect integer answers between 0 and 999.

Extract answer from solution text. Args: solution: The solution text to extract answer from problem_type: Type of problem (g

Extract Python code blocks from text.

Get response from the LLM with retry logic.

Generate appropriate prompt based on category.

Get all strategies for a specific problem type.

Initialize and cache all server capabilities

Check if running on Apple Silicon

Load existing results from file if it exists.

Load existing results from file if it exists.