Verify a single solution with multiple passes

Write a log entry to the appropriate JSONL file

Try to acquire a slot for this provider. Returns True if acquired, False if timeout or no limit.

Add an example to the strategy.

Add an example to a strategy.

Add a reasoning example to a strategy.

Add a new strategy to the database.

Add response to cache with metadata

Add a verification result to the workspace

Aggregate multiple paths based on their confidence scores.

Analyze the current draft to identify gaps, weaknesses, and areas needing research This guides the next retrieval iteration (draft-gu

Analyze and print summary statistics of the results. Args: results (list[Dict]): List of evaluation results

Analyze and print comprehensive statistics

Analyze and print comprehensive statistics of IMO evaluation results

Analyze and print comprehensive statistics with full credit prioritized

Analyze and print summary statistics of the results. Args: results (List[Dict]): List of evaluation results n (int): Num

Augment the system prompt with selected strategies and reasoning examples. Instructs the LLM to apply the strategies in its solution.

Calculate the confidence score (Δ) as specified in the paper. Args: logits: List of logits for each decoding step answer

Calculate log probabilities for a sequence of tokens

Calculate full credit scores for various subsets (Novel, IMO 2024, USAMO 2025) Returns dictionary with subset names and their (solved, total,

Calculate token entropy: H = -∑P(j) log P(j) Args: logits: Raw logits tensor for current token position

Mock completions.create with realistic IMO25 responses

Check if extracted answer matches the golden answer for the problem

Check if consensus has been reached among traces. Args: traces: List of trace results Retur

Splits a long document into token-limited chunks based on a separator, ensuring each chunk fits within `chunk_size`. Uses a greedy approach

Classify query complexity. Args: query: The query to classify Returns: Tuple of

Use the LLM to classify the problem type, ensuring the result is one of the valid types. Args: content: The query/problem to cla

Remove any remaining placeholder tags from the final report. This is a final cleanup step to ensure no incomplete research t

Close the browser session

Close and remove a session.

Collapses context chunk pairs in sliding window until the total token count fits within the context window. Args: client: LLM API cl

Compare the correct answer with the predicted answer.

Compare predicted answer with ground truth Uses both exact match and semantic equivalence

Compute cosine similarity between two texts using TF-IDF vectorization. This is a local implementation that doesn't require any external API.

Construct prompt based on split type.

Implement CoT-decoding for a given chat input. Args: model: The Hugging Face transformer model. tokenizer: The associate

Count tokens within <think>...</think> tags in the given text. Args: text: The text to analyze tokenizer: Optional token

Count the number of tokens in a text string.

Create the complete benchmark dataset

Create a prompt for comparing candidate solutions. Args: candidates: List of candidate responses query: The original use

Factory function to create an inference client

Create an MLX inference pipeline

Pre-tokenize context patterns for efficient matching during generation. Similar to how guided mode does token-based matching.

Decompose complex research query into focused sub-queries This implements the query planning phase of TTD-DR

Main DeepConf decoding function for integration with OptILLM. Implements confidence-aware reasoning with early termination for local mod

Core denoising step: integrate retrieved information with current draft This is the heart of the diffusion process

Detect whether this is a code, math, or generic problem

Stage 1: Discover reasoning structure for the given task. Args: task_description: Description of the task type

Download and parse the AnswerBench CSV dataset

Download SimpleQA dataset if not cached

Download and parse the ProofBench CSV dataset

Evaluate the dataset using RTC methodology.

Evaluate the quality improvement of the current draft vs previous iteration Used for termination decisions and component fitness upda

Evaluate a model on the dataset using a specific approach. Returns metrics and detailed results.

Evaluate if any of the n attempts got the correct answer. Args: attempts (List[Dict]): List of attempt results correct_a

Evaluate a single question

IMO25-style evaluation using rigorous two-stage verification system: 1. Detailed verification with comprehensive IMO grader prompt 2. Sim

Evaluate how effective each strategy was in generating the response. Args: response: The LLM's final response to the query

Attempt to execute the code using Jupyter notebook kernel and return result or error.

Scan text (with Markdown/LaTeX wrappers intact) and return 'A', 'B', 'C', or 'D' if a correct-answer declaration is found. Otherwise retu

Extract the answer from a math solution response.

Universal answer extraction using math-verify library with fallback patterns. Args: solution: The solution text to extra

Extract the final answer from a solution

Extract the final answer from a solution using unified answer extraction

Extract the index of the correct answer from a multiple-choice question. Args: question: The question text containing choices

Extract clean final answer from MARS synthesis text Args: text: Full synthesis output with reasoning mode: 'auto', 'code', '

Extract clean code from synthesis output Finds the last complete code block as the final answer

Extract and verify the final answer using official IMO 2025 solutions

Try to extract just the final answer from a response. This is generic and looks for common patterns.

Extract the content from the first turn in the conversation.

Extract answer for generic (non-code, non-math) problems Returns the last paragraph or sentence as the final answer For proof-based probl

Extract text content and finish reason from an LLM response. Supports both non-streaming responses (dict-like with `.choices[0].message.cont

Extract clean math answer from synthesis output Finds the last \\boxed{} answer as the final answer

Extract schema from response_format field.