A set of parquet files.

Run SQL query against the outputs of input_deps.

All inputs of a logical plan are represented as `DataSourceNode`. It does not depend on any other node.

The logical plan that defines a directed acyclic computation graph.

Global context for each logical plan. Right now it's mainly used to keep a list of Python UDFs.

Information about a partition of a dataset.

A distributed data collection. It represents a 2 dimensional table of rows and columns. Internally, it's a wrapper around a `Node` and a `Se

Partition the outputs of `input_deps` into n partitions based on the hash values of `hash_columns`.

Collect the output files of `input_deps` to `output_path`. Depending on the options, it may create hard links, symbolic links, manifest files

Evenly distribute the output files or rows of `input_deps` into n partitions.

Select columns from output of an input node.

Use a background thread to iterate over an iterable. Examples -------- The following code snippet is a common pattern to read record

A helper class that includes boilerplate code to execute a logical plan.

Run user-defined code to process the input datasets as a series of arrow tables.

Run Python code to process the input datasets, which have been loaded as Apache Arrow tables. See https://arrow.apache.org/docs/python/genera

Create an execution plan (tasks) from a logical plan (nodes).

Run Python code to process the input datasets with `PythonScriptNode.process(...)`. If the code needs to access attributes of runtime task,

Consolidate partitions into larger ones.

Load existing partitioned dataset (only parquet files are supported).

The configuration and state for a running job.

A set of csv files.

Run Python code to process the input datasets, which have been loaded as RecordBatchReader. See https://arrow.apache.org/docs/python/generate

An arrow table.

A unique identifier for a job.

A virtual node that assembles multiple nodes and outputs nothing.

Union two or more nodes into one flow of data.

A set of files.

A set of json files.

Run Python code to process the input datasets as a series of pandas DataFrames.

A range of rows in a file.

The task scheduler.

A unique identifier for a task.

A unique identifier for a task at runtime. Besides the task id, it also includes the epoch and retry count.

Configuration for a session.

A directed acyclic graph (DAG) of tasks.

The task executor.

Intercept standard logging messages toward loguru sinks. See https://loguru.readthedocs.io/en/stable/overview.html#entirely-compatible-with-s

Limit the number of rows of the output of an input node.

A unique identifier for each node.

Optimize the logical plan.

Run Python code to process the input datasets as a single pandas DataFrame.

A pandas dataframe.

A dataset that is partitioned into multiple datasets.

Performance statistics for a task.

Base class for all platforms.

A state observer that push the scheduler state into a queue when finished.