Save the qa and corpus to the AutoRAG compatible parquet file. It is not for the data creation, for running AutoRAG. If you want to save it dir

Get asyncio event loop safely.

Pop parameters from the given func and return them. It automatically deletes the parameters like "self" or "cls". :param func: The function to po

Processes tasks in batches asynchronously. :param tasks: A list of no-argument functions or coroutines to be executed. :param batch_size: The num

Load a summary file from summary_path. :param summary_path: The path of the summary file. :param dict_columns: The columns that are dictionary ty

Recursively convert collections to Python lists.

Ingest given corpus data to the vectordb. It truncates corpus content when the embedding model is OpenAIEmbedding to the 8000 tokens. Plus, when t

Recursively apply a function to all elements in a list, tuple, set, np.ndarray, or pd.Series and return as List. :param func: Function to apply to

Method for measuring execution speed of the function.

Filter results by value's threshold. :param results: The result list to be filtered. :param value: The value list to be filtered. It must ha

Make a batch of elems with batch_size.

Sample the corpus for making QA. It selects the subset of the corpus and makes QA set from it. You can generate questions from the created ques

Turn metrics to list of metric names and parameter list. :param metrics: List of string or dictionary. :return: The list of metric names and dic

This function flattens the input list and applies the function to the elements. After that, it reconstructs the list to the original shape. Its sp

Parse documents to use langchain document_loaders(parse) method :param data_path_list: The list of data paths to parse. :param parse_method: A la

Start AutoRAG trial. The trial means one experiment to optimize the RAG pipeline. It consists of ingesting corpus data, running all nodes and m

Recursively find all values for a specific key in a nested dictionary or list. :param data: The dictionary or list to search. :param target_key:

Load Runner from the evaluated trial folder. Must already be evaluated using Evaluator class. It sets the project_dir as the parent directory o

Taken from the official evaluation script for v1.1 of the SQuAD dataset. Lower text and remove punctuation, articles, and extra whitespace.

Explode index_values and explode_values. The index_values and explode_values must have the same length. It will flatten explode_values and keep in

Make combinations from target_dict. The target_dict key value must be a string, and the value can be a list of values or single value. If generat

Generates Embeddings with input validation and retry mechanism. Args: sentences: Texts or Sentences to embed prompt_name: The name o

Extract the optimal pipeline from the evaluated trial. :param trial_path: The path to the trial directory that you want to extract the pipeline fr

Chunk texts from the parsed result to use llama index chunk method :param texts: The list of texts to chunk from the parsed result :param chunker

Load a YAML configuration file for AutoRAG. It contains safe loading, converting string to tuple, and insert environment variables. :param yaml_p

Evaluate retrieval node from retrieval node result dataframe. :param result_df: The result dataframe from a retrieval node. :param metric_inputs:

This function extract values from nodes' modules' module_param. :param nodes: The nodes you want to extract values from. :param key: The key of m

Calculate the highest similarity scores between query embeddings and content embeddings. :param query_embeddings: A list of lists containing query

Hybrid CC function. CC (convex combination) is a method to fuse lexical and semantic retrieval results. It is a method that first normalizes the s

Chunk texts from the parsed result to use langchain chunk method :param texts: The list of texts to chunk from the parsed result :param chunker: