Add a command-line argument. This is a wrapper of `argparse.ArgumentParser.add_argument(...)`.

Start a scheduler and `num_executors` executors to execute `plan`, and wait the execution completed, then assert if it succeeds.

Partition the outputs of `input_deps` into n partitions. Parameters ---------- npartitions The number of partitions. The inp

Start or stop the timer. If `metric_name` is provided, add the elapsed time to the task's performance metrics. Example: ```

Load the dataset to an arrow table. Parameters ---------- max_workers, optional The maximum number of wo

Apply a function to each row. Parameters ---------- sql_or_func A SQL expression or a function to apply

The base class for all nodes in logical plan. Parameters ---------- ctx The context of logical plan.

Convert to an arrow Table. This operation will trigger execution of the lazy transformations performed on this DataFrame.

Create a DataFrame from a list of local Python objects.

Split the dataset into a list of partitioned datasets.

This visitor calls visit() on all children of the node.

Dump an object to a file. Args: obj: The object to dump. path: The path to the file to dump the object to. buffering: The

Repartition the data into the given number of partitions. Parameters ---------- npartitions The dataset

Create a DataFrame from a pyarrow Table.

The entry point for executor and scheduler of `plan`.

Evenly split items into npartitions. Example:: >>> split_into_rows(list(range(10)), 3) [[0, 1, 2, 3], [4, 5, 6], [7, 8, 9]]

Construct a dataset from a list of paths. Parameters ---------- paths A path or a list of paths or path

Compute the data and return the datasets.

Load an object from a file.

Return all rows. This operation will trigger execution of the lazy transformations performed on this DataFrame.

Count the number of rows. If this dataframe consists of more than a read, or if the row count can't be determined from the m

Shutdown the session.

Return a sql fragment that represents the dataset.

Wait for all DataFrames to be computed. Example ------- This can be used to wait for multiple outputs from a pipelin

Create a DataFrame from a pandas DataFrame.

Reset the dataset with new paths, root_dir, and recursive flag.

Return an arrow record batch reader to read the dataset. Parameters ---------- batch_size, optional The

Log the dataset to the logger.

Merge multiple datasets into a single dataset.

Execute a SQL query on each partition of the input DataFrames. The query can contain placeholder `{0}`, `{1}`, etc. for the input Da

Evenly split into `npartition` datasets by files.

Create a DataFrame from CSV files.

Parse size string to bytes. e.g. 1k -> 1024, 1M -> 1024^2, 1G -> 1024^3, 1T -> 1024^4

Convert the dataset to a pandas dataframe.

Compute the data. This operation will trigger execution of the lazy transformations performed on this DataFrame.

Define a Python UDF to be referenced in the logical plan. Currently only scalar functions (return one element per row) are supported.