MCPcopy Create free account

hub / github.com/deepseek-ai/smallpond / functions

Functions992 in github.com/deepseek-ai/smallpond

⨍ Functions 992◇ Types & classes 154↳ Endpoints 2
↓ 1 callersMethodprocess
( self, runtime_ctx: RuntimeContext, input_tables: List[arrow.Table] )
smallpond/execution/task.py:2497
↓ 1 callersMethodprocess
( self, runtime_ctx: RuntimeContext, input_dfs: List[pd.DataFrame] )
smallpond/execution/task.py:2515
↓ 1 callersMethodprocess
( self, runtime_ctx: RuntimeContext, input_dfs: List[pd.DataFrame] )
smallpond/execution/task.py:2529
↓ 1 callersMethodprocess_batch
( self, batch_index: int, input_datasets: List[DataSet], conn: duckdb.DuckDBPy
smallpond/execution/task.py:2649
↓ 1 callersFunctionproducer
(wq: WorkQueue, id: int, numItems: int, numConsumers: int)
tests/test_workqueue.py:30
↓ 1 callersMethodrandom_shuffle
Randomly shuffle all rows globally.
smallpond/dataframe.py:377
↓ 1 callersMethodrandom_uint32
(self)
smallpond/execution/task.py:856
↓ 1 callersMethodrelease_gpu
Release GPU resources to the pool.
smallpond/execution/executor.py:333
↓ 1 callersMethodreset_state
(self, current_epoch: int)
smallpond/execution/scheduler.py:243
↓ 1 callersMethodrestore_input_state
( self, runtime_state: RuntimeState, input_readers: List[arrow.RecordBatchReader] )
smallpond/execution/task.py:2241
↓ 1 callersMethodrun
( self, plan: LogicalPlan, job_id: Optional[str] = None, job_time: Optional[fl
smallpond/execution/manager.py:66
↓ 1 callersMethodrun
(self)
smallpond/execution/workqueue.py:154
↓ 1 callersFunctionrun_executor
(runtime_ctx: RuntimeContext, executor: Executor)
tests/test_fabric.py:39
↓ 1 callersMethodsave_task_final_state
(self, finished_task: Task)
smallpond/execution/scheduler.py:699
↓ 1 callersMethodsched_loop
(self, pool: ThreadPoolExecutor)
smallpond/execution/scheduler.py:1103
↓ 1 callersMethodset_memory_limit
(self, soft_limit: int, hard_limit: int)
smallpond/execution/task.py:919
↓ 1 callersMethodskip_probes
(self, epochs: int)
smallpond/execution/executor.py:148
↓ 1 callersMethodspawn
Return an instance of subclass of `PythonScriptTask`. The subclass should override `PythonScriptTask.process(...)`. Examples
smallpond/logical/node.py:523
↓ 1 callersMethodspawn
Return an instance of subclass of `ArrowComputeTask`. The subclass should override `ArrowComputeTask.process(...)`. Examples
smallpond/logical/node.py:686
↓ 1 callersMethodspawn
Return an instance of subclass of `ArrowStreamTask`. The subclass should override `ArrowStreamTask.process(...)`. Examples -
smallpond/logical/node.py:865
↓ 1 callersMethodspawn
(self, *args, **kwargs)
smallpond/logical/node.py:1176
↓ 1 callersMethodsplit_string
(self, s: str)
smallpond/contrib/warc.py:108
↓ 1 callersMethodsql_query_fragment
( self, filesystem: fsspec.AbstractFileSystem = None, conn: duckdb.DuckDBPyConnection
smallpond/logical/dataset.py:1015
↓ 1 callersMethodstart
(self, pool: ThreadPoolExecutor)
smallpond/execution/scheduler.py:341
↓ 1 callersMethodstart_speculative_execution
(self)
smallpond/execution/scheduler.py:592
↓ 1 callersMethodstop_job
Stop the job.
smallpond/platform/base.py:48
↓ 1 callersMethodstop_running_tasks
(self, task_key: str)
smallpond/execution/scheduler.py:729
↓ 1 callersMethodsuspend_good_executors
(self)
smallpond/execution/scheduler.py:563
↓ 1 callersMethodto_arrow_table
( self, max_workers=16, filesystem: fsspec.AbstractFileSystem = None, conn: du
smallpond/logical/dataset.py:422
↓ 1 callersMethodto_arrow_table
( self, max_workers=16, filesystem: fsspec.AbstractFileSystem = None, conn: du
smallpond/logical/dataset.py:1075
↓ 1 callersMethodto_batch_reader
( self, batch_size=DEFAULT_BATCH_SIZE, filesystem: fsspec.AbstractFileSystem = None,
smallpond/logical/dataset.py:781
↓ 1 callersMethodtry_boost_resource
(self, item: WorkItem, executor: RemoteExecutor)
smallpond/execution/scheduler.py:753
↓ 1 callersMethodtry_relax_memory_limit
(self, task: Task, executor: RemoteExecutor)
smallpond/execution/scheduler.py:742
↓ 1 callersMethodupdate
(self, sched_state: Scheduler)
smallpond/execution/manager.py:25
↓ 1 callersMethodupdate_batch_offsets
(self, batch_indices: Optional[List[int]])
smallpond/execution/task.py:2108
↓ 1 callersMethodupdate_executor_states
(self)
smallpond/execution/scheduler.py:661
↓ 1 callersMethodupdate_queue
(self)
smallpond/execution/executor.py:79
↓ 1 callersMethodupdate_state
(self, current_epoch: int)
smallpond/execution/scheduler.py:246
↓ 1 callersMethodvisit
Visit a node depending on its type. If the method for the node type is not implemented, call `generic_visit`.
smallpond/logical/node.py:2006
↓ 1 callersMethodvisit_arrow_compute_node
(self, node: ArrowComputeNode, depth: int)
smallpond/logical/node.py:2064
↓ 1 callersMethodvisit_arrow_stream_node
(self, node: ArrowStreamNode, depth: int)
smallpond/logical/node.py:2067
↓ 1 callersMethodvisit_consolidate_node
(self, node: ConsolidateNode, depth: int)
smallpond/logical/node.py:2055
↓ 1 callersMethodvisit_data_sink_node
(self, node: DataSinkNode, depth: int)
smallpond/logical/node.py:2046
↓ 1 callersMethodvisit_data_source_node
(self, node: DataSourceNode, depth: int)
smallpond/logical/node.py:2043
↓ 1 callersMethodvisit_limit_node
(self, node: LimitNode, depth: int)
smallpond/logical/node.py:2076
↓ 1 callersMethodvisit_partition_node
(self, node: PartitionNode, depth: int)
smallpond/logical/node.py:2058
↓ 1 callersMethodvisit_projection_node
(self, node: ProjectionNode, depth: int)
smallpond/logical/node.py:2073
↓ 1 callersMethodvisit_python_script_node
(self, node: PythonScriptNode, depth: int)
smallpond/logical/node.py:2061
↓ 1 callersMethodvisit_query_engine_node
(self, node: SqlEngineNode, depth: int)
smallpond/logical/node.py:2070
↓ 1 callersMethodvisit_root_node
(self, node: RootNode, depth: int)
smallpond/logical/node.py:2049
↓ 1 callersMethodvisit_union_node
(self, node: UnionNode, depth: int)
smallpond/logical/node.py:2052
↓ 1 callersMethodwrite_flat_partitions
( self, conn: duckdb.DuckDBPyConnection, batch_index: int, input_dataset: Parq
smallpond/execution/task.py:3115
↓ 1 callersMethodwrite_hive_partitions
( self, conn: duckdb.DuckDBPyConnection, batch_index: int, input_dataset: Parq
smallpond/execution/task.py:3082
↓ 1 callersMethodwrite_parquet_lazy
Write data to a series of parquet files under the given path. This is a non-blocking operation. See :func:`write_parquet` for a bloc
smallpond/dataframe.py:630
Method__call__
(self, *args, **kwargs)
smallpond/logical/udf.py:242
Method__call__
(self, *args: Any, **kwds: Any)
smallpond/execution/task.py:660
Method__call__
(self, *args: Any, **kwds: Any)
smallpond/execution/scheduler.py:434
Method__call__
(self, sched_state: Scheduler)
tests/test_fabric.py:53
Method__enter__
(self)
smallpond/utility.py:123
Method__enter__
(self)
smallpond/execution/executor.py:68
Method__eq__
(self, other: "PartitionInfo")
smallpond/execution/task.py:529
Method__exit__
(self, exc_type, exc_value, traceback)
smallpond/utility.py:127
Method__exit__
(self, exc_type, exc_value, traceback)
smallpond/execution/executor.py:71
Method__getattr__
(self, name)
smallpond/utility.py:82
Method__getitem__
(self, key: int)
smallpond/logical/dataset.py:396
Method__getstate__
(self)
smallpond/execution/scheduler.py:287
Method__getstate__
(self)
smallpond/execution/scheduler.py:437
Method__hash__
(self)
smallpond/execution/task.py:532
Method__init__
(self, base_obj: Any)
smallpond/utility.py:73
Method__init__
(self, iterable: Iterable, max_buffer_size=1)
smallpond/utility.py:113
Method__init__
(self, **kwargs)
smallpond/dataframe.py:28
Method__init__
(self, session: Session, plan: Node, recompute: bool = False)
smallpond/dataframe.py:216
Method__init__
Create a smallpond environment.
smallpond/session.py:30
Method__init__
( self, ctx: Context, input_deps: Tuple[Node, ...], num_rows=200, **kwargs )
smallpond/contrib/log_dataset.py:28
Method__init__
(self, runtime_ctx: RuntimeContext)
smallpond/logical/planner.py:12
Method__init__
(self, exclude_nodes: Set[Node])
smallpond/logical/optimizer.py:10
Method__init__
(self)
smallpond/logical/node.py:85
Method__init__
Construct a DataSourceNode. See :func:`Node.__init__` to find comments on other parameters. Parameters ---------- da
smallpond/logical/node.py:308
Method__init__
Construct a DataSinkNode. See :func:`Node.__init__` to find comments on other parameters. Parameters ---------- outp
smallpond/logical/node.py:347
Method__init__
Construct a PythonScriptNode. See :func:`Node.__init__` to find comments on other parameters. Parameters ----------
smallpond/logical/node.py:465
Method__init__
Construct a ArrowComputeNode. See :func:`Node.__init__` to find comments on other parameters. Parameters ----------
smallpond/logical/node.py:591
Method__init__
Construct a ArrowStreamNode. See :func:`Node.__init__` to find comments on other parameters. Parameters ---------- p
smallpond/logical/node.py:747
Method__init__
Construct a SqlEngineNode. See :func:`Node.__init__` to find comments on other parameters. Parameters ---------- sql
smallpond/logical/node.py:966
Method__init__
Union two or more `input_deps` into one flow of data. Parameters ---------- input_deps All input deps sh
smallpond/logical/node.py:1189
Method__init__
Assemble multiple nodes into a root node.
smallpond/logical/node.py:1206
Method__init__
Effectively reduces the number of partitions without shuffling the data across the network. Parameters ---------- di
smallpond/logical/node.py:1218
Method__init__
Partition the outputs of `input_deps` into n partitions. Parameters ---------- npartitions The dataset w
smallpond/logical/node.py:1257
Method__init__
( self, ctx: Context, input_deps: Tuple[Node, ...], npartitions: int,
smallpond/logical/node.py:1356
Method__init__
( self, ctx: Context, input_deps: Tuple[Node, ...], npartitions: int,
smallpond/logical/node.py:1399
Method__init__
( self, ctx: Context, partitioned_datasets: List[DataSet], dimension: str = None )
smallpond/logical/node.py:1444
Method__init__
Evenly distribute the output files or rows of `input_deps` into n partitions. Parameters ---------- partition_by_row
smallpond/logical/node.py:1470
Method__init__
( self, ctx: Context, input_deps: Tuple[Node, ...], npartitions: int,
smallpond/logical/node.py:1542
Method__init__
Construct a HashPartitionNode. See :func:`Node.__init__` to find comments on other parameters. Parameters ----------
smallpond/logical/node.py:1665
Method__init__
( self, ctx: Context, input_deps: Tuple[Node, ...], npartitions: int,
smallpond/logical/node.py:1805
Method__init__
( self, ctx: Context, input_deps: Tuple[Node, ...], split_points: List,
smallpond/logical/node.py:1856
Method__init__
Construct a ProjectNode to select only the `columns` from output of `input_dep`. Parameters ---------- input_dep
smallpond/logical/node.py:1887
Method__init__
Construct a LimitNode to limit the number of rows of the output of `input_dep`. Parameters ---------- input_dep
smallpond/logical/node.py:1971
Method__init__
(self, ctx: Context, root_node: Node)
smallpond/logical/node.py:2085
Method__init__
(self, fields: Union[Dict[str, str], List[str]])
smallpond/logical/udf.py:108
Method__init__
(self, child)
smallpond/logical/udf.py:122
← previousnext →301–400 of 992, ranked by callers