Create an empty file for a partition according to the schema of the dataset. Return the path relative to the output directory.

Read record batches from a range of a parquet file.

Build a batch reader from a list of row ranges.

Generate a new node id.

Get the partition key of the parquet file. Examples -------- ``` >>> ParquetDataSet._read_partition_key("out

Spawn a new job to run the current script.

Start grafana server if it exists.

Start prometheus server if it exists.

Acquire GPU resources with the given quota. Return a dict of granted GPUs with their quotas. `release_gpu` should be called l

Check if the expected data matches the actual data. Raise an error if there is a mismatch.

Called after run() even if there is an exception.

Convert all string and binary types to large types recursively.

Get the default memory allocator for the platform.

Whether to share log analytics by default.

Dump the logical plan graph to a file.

Dump the task timeline to a file.

Called after run() to finalize the processing.

Load config from arguments and environment variables. If not specified, use the default value.

Generate a list of long paths

Generate random data

Generate multiple parquet files with a specified number of random data lines

Generate multiple files, each containing a specified number of random data lines

Return the homepath of grafana.

Get the logical plan graph.

Return a graphviz graph that shows the structure of the logical plan.

Called before run() to prepare for running the task.

Check if the data is ready on disk.

Apply the given function to batches of data. Parameters ---------- func A function or a callable class t

This method can be overridden in subclass of `PythonScriptTask`. Parameters ---------- runtime_ctx The r

This method can be overridden in subclass of `ArrowComputeTask`. Parameters ---------- runtime_ctx The r

This method can be overridden in subclass of `ArrowStreamTask`. Parameters ---------- runtime_ctx The ru