MCPcopy Create free account

hub / github.com/deepseek-ai/smallpond / functions

Functions992 in github.com/deepseek-ai/smallpond

⨍ Functions 992◇ Types & classes 154↳ Endpoints 2
↓ 3 callersFunctionexecute_command
(cmd: str, env: Dict[str, str] = None, shell=False)
smallpond/utility.py:45
↓ 3 callersMethodfilter
Filter out rows that don't satisfy the given predicate. Parameters ---------- sql_or_func A SQL expressi
smallpond/dataframe.py:423
↓ 3 callersMethodfinalize
(self)
smallpond/execution/task.py:964
↓ 3 callersMethodflat_map
Apply a function to each row and flatten the result. Parameters ---------- sql_or_func A SQL expression
smallpond/dataframe.py:529
↓ 3 callersMethodget_retry_task
(self, key: str)
smallpond/execution/scheduler.py:784
↓ 3 callersMethodget_runnable_tasks
(self, finished_task: Task)
smallpond/execution/scheduler.py:715
↓ 3 callersFunctiongray_sort_benchmark
( record_nbytes, key_nbytes, total_data_nbytes, gensort_batch_nbytes, num_data_partitions,
benchmarks/gray_sort_benchmark.py:265
↓ 3 callersMethodinitialize
(self)
smallpond/execution/task.py:926
↓ 3 callersMethodinitialize
(self)
tests/test_execution.py:60
↓ 3 callersMethodlimit
Limit the number of rows to the given number. Unlike `take`, this method doesn't trigger execution.
smallpond/dataframe.py:607
↓ 3 callersMethodnew
()
smallpond/execution/task.py:99
↓ 3 callersMethodnotify_state_observers
(self, force_notify=False)
smallpond/execution/scheduler.py:939
↓ 3 callersMethodrandom_float
(self)
smallpond/execution/task.py:859
↓ 3 callersMethodread_parquet
Create a DataFrame from Parquet files.
smallpond/dataframe.py:48
↓ 3 callersMethodset_current_task
(self, task: "Task" = None)
smallpond/execution/task.py:317
↓ 3 callersMethodslim_copy
(self)
smallpond/logical/node.py:269
↓ 3 callersMethodstart_job
( self, num_nodes: int, entrypoint: str, args: List[str], envs: dict =
smallpond/platform/mpi.py:18
↓ 3 callersMethodstop
(self)
smallpond/execution/scheduler.py:347
↓ 3 callersMethodstop_executors
(self)
smallpond/execution/scheduler.py:588
↓ 3 callersMethodto_pandas
Convert to a pandas DataFrame. This operation will trigger execution of the lazy transformations performed on this DataFrame.
smallpond/dataframe.py:695
↓ 3 callersMethodupdate
(self, sched_state: "Scheduler")
smallpond/execution/scheduler.py:377
↓ 3 callersMethodwrite_parquet
Write data to a series of parquet files under the given path. This is a blocking operation. See :func:`write_parquet_lazy` for a non
smallpond/dataframe.py:616
↓ 2 callersMethod_check_partition_datasets
( self, orig_dataset: ParquetDataSet, partition_func, npartition )
tests/test_dataset.py:44
↓ 2 callersMethod_close_file_writers
(self)
smallpond/execution/task.py:2843
↓ 2 callersMethod_generate_parquet_dataset
( self, output_path, npartitions, num_rows, row_group_size )
tests/test_dataset.py:33
↓ 2 callersMethod_get_or_create_tasks
Get or create tasks to compute the data.
smallpond/dataframe.py:228
↓ 2 callersMethod_init_file_partitions
Return `npartition` empty datasets.
smallpond/logical/dataset.py:331
↓ 2 callersMethod_init_logs
(self, exec_id: str, capture_stdout_stderr: bool = False)
smallpond/execution/task.py:377
↓ 2 callersMethod_list_works
(self, path: str, expand_batch=True)
smallpond/execution/workqueue.py:315
↓ 2 callersMethod_push_unbuffered
(self, item: WorkItem)
smallpond/execution/workqueue.py:238
↓ 2 callersMethod_wait_pending_writes
(self)
smallpond/execution/task.py:2837
↓ 2 callersMethod_wrap_output
( self, output: Union[arrow.Table, StreamOutput], batch_indices: List[int] = None )
smallpond/execution/task.py:2204
↓ 2 callersMethod_write_to_partition
( self, partition_idx, partition, pending_write: Future = None )
smallpond/execution/task.py:2897
↓ 2 callersMethodadjust_row_group_size
( self, nbytes, num_rows, max_row_group_size=MAX_ROW_GROUP_SIZE, max_r
smallpond/execution/task.py:880
↓ 2 callersMethodbind
(self, conn: duckdb.DuckDBPyConnection)
smallpond/logical/udf.py:157
↓ 2 callersFunctionclamp_value
(val, minval, maxval)
smallpond/common.py:36
↓ 2 callersFunctioncombine_buffered_batches
( batches: List[arrow.RecordBatch], )
smallpond/io/arrow.py:249
↓ 2 callersMethodcompute_avg_row_size
(self, nbytes, num_rows)
smallpond/execution/task.py:877
↓ 2 callersFunctionconvert_types_to_large_string
Convert all string and binary types to large types in the schema.
smallpond/io/arrow.py:124
↓ 2 callersFunctioncprofile_to_string
( perf_profile: cProfile.Profile, order_by=pstats.SortKey.TIME, top_k=20 )
smallpond/utility.py:62
↓ 2 callersMethodcreate
(ctx: RuntimeContext, id: str)
smallpond/execution/scheduler.py:301
↓ 2 callersMethodcreate_consumer_task
( self, runtime_ctx: RuntimeContext, input_deps: List[Task], partition_infos:
smallpond/logical/node.py:1320
↓ 2 callersMethodcreate_driver
(self, num_executors: int)
tests/test_driver.py:19
↓ 2 callersMethodcreate_duckdb_extension
Load a DuckDB extension.
smallpond/logical/node.py:148
↓ 2 callersMethodcreate_external_module
Load an external DuckDB module.
smallpond/logical/node.py:140
↓ 2 callersMethodcreate_producer_task
( self, runtime_ctx: RuntimeContext, input_deps: List[Task], partition_infos:
smallpond/logical/node.py:1311
↓ 2 callersMethoddefault_data_root
Get the default data root for the platform. If the platform does not have a default data root, return None.
smallpond/platform/base.py:69
↓ 2 callersMethoddefault_job_id
Return the default job id.
smallpond/platform/base.py:55
↓ 2 callersMethoddefault_job_time
Return the default job time.
smallpond/platform/base.py:62
↓ 2 callersMethoddump_output
(self, output_iter: Iterable[StreamOutput])
smallpond/execution/task.py:2352
↓ 2 callersFunctionexec_task
(task: Task, output_path: Optional[str])
smallpond/execution/task.py:3627
↓ 2 callersMethodexplain_str
Return a string that shows the structure of the logical plan.
smallpond/logical/node.py:2092
↓ 2 callersFunctionfile_io_benchmark
( input_paths, npartitions, io_engine="duckdb", batch_size=DEFAULT_BATCH_SIZE, row_group_s
benchmarks/file_io_benchmark.py:14
↓ 2 callersFunctionfilter_schema
( schema: arrow.Schema, included_cols: Optional[List[str]] = None, excluded_cols: Optional[List[st
smallpond/io/arrow.py:143
↓ 2 callersMethodflush
(self)
smallpond/execution/scheduler.py:231
↓ 2 callersMethodflush
(self)
smallpond/execution/workqueue.py:260
↓ 2 callersFunctionfstest
( sp: Session, input_path: Optional[str], output_path: Optional[str], size: Optional[str],
examples/fstest.py:139
↓ 2 callersFunctiongenerate_data
Generate all data for testing.
tests/datagen.py:160
↓ 2 callersFunctiongenerate_data
Generate data for the slice [offset, offset + length). The full data is a repeated sequence of [0x00000000, 0x00000001, ..., 0xffffffff] in l
examples/fstest.py:88
↓ 2 callersFunctiongenerate_random_records
( ctx, record_nbytes, key_nbytes, total_data_nbytes, gensort_batch_nbytes, num_data_pa
benchmarks/gray_sort_benchmark.py:206
↓ 2 callersFunctiongenerate_random_string
Generate a random string of a specified length
tests/datagen.py:98
↓ 2 callersFunctiongenerate_random_url
Generate a random URL
tests/datagen.py:103
↓ 2 callersMethodget_perf_stats
(self, name)
smallpond/logical/node.py:286
↓ 2 callersFunctionget_platform
Get a platform by name. If name is not specified, try to get an available platform.
smallpond/platform/__init__.py:11
↓ 2 callersFunctionhas_magic
(s)
smallpond/logical/dataset.py:36
↓ 2 callersFunctionhash_partition_benchmark
( input_paths, npartitions, hash_columns, engine_type="duckdb", use_parquet_writer=False,
benchmarks/hash_partition_benchmark.py:15
↓ 2 callersMethodinitialize
(self, exec_id: str, root_exist_ok=True, cleanup_root=False)
smallpond/execution/task.py:327
↓ 2 callersFunctioniter_io_slice
Generate the IO (offset, size) for the slice [offset, offset + length) with the given block size. `block_size` can be an integer or a range [
examples/fstest.py:102
↓ 2 callersMethodjoin_running_procs
(self, timeout=30)
tests/test_fabric.py:117
↓ 2 callersMethodload_input_batch
( self, conn: duckdb.DuckDBPyConnection, batch_index: int, input_dataset: Parq
smallpond/execution/task.py:3047
↓ 2 callersMethodmerge_metrics
(self, metrics: Dict[str, int])
smallpond/execution/task.py:1103
↓ 2 callersFunctionon_hf3fs
(path: str)
smallpond/io/filesystem.py:18
↓ 2 callersMethodpartial_sort
Sort rows by the given columns in each partition. Parameters ---------- by A column or a list of columns
smallpond/dataframe.py:397
↓ 2 callersMethodpartition_by_rows
Evenly split the dataset into `npartition` partitions by rows. If `random_shuffle` is True, shuffle the files before partitioning.
smallpond/logical/dataset.py:807
↓ 2 callersMethodpartition_by_rows
Evenly split a list of row ranges into `npartition` partitions.
smallpond/io/arrow.py:65
↓ 2 callersMethodpartition_by_size
Split the dataset into multiple partitions so that each partition has at most `max_partition_size` bytes.
smallpond/logical/dataset.py:844
↓ 2 callersMethodpop_running_work
(self, key: str)
smallpond/execution/scheduler.py:184
↓ 2 callersMethodprocess_finished_tasks
(self, pool: ThreadPoolExecutor)
smallpond/execution/scheduler.py:1250
↓ 2 callersMethodremove_empty_files
Remove empty parquet files from the dataset.
smallpond/logical/dataset.py:963
↓ 2 callersMethodreset
(self)
smallpond/execution/task.py:2123
↓ 2 callersMethodrun
(self)
smallpond/execution/scheduler.py:985
↓ 2 callersMethodrun_on_ray
Run the task on Ray. Return an `ObjectRef`, which can be used with `ray.get` to wait for the output dataset.
smallpond/execution/task.py:1107
↓ 2 callersMethodsetUp
(self)
tests/test_workqueue.py:69
↓ 2 callersMethodshared_log_root
Return the shared log root.
smallpond/platform/base.py:88
↓ 2 callersFunctionshuffle_data
( input_paths, num_out_data_partitions: int = 0, num_data_partitions: int = 10, num_hash_parti
examples/shuffle_data.py:14
↓ 2 callersFunctionshuffle_mock_urls
( input_paths, npartitions: int = 10, sort_rand_keys=True, engine_type="duckdb" )
examples/shuffle_mock_urls.py:14
↓ 2 callersFunctionsort_mock_urls_v2
( sp: Session, input_paths: List[str], output_path: str, npartitions: int )
examples/sort_mock_urls_v2.py:8
↓ 2 callersFunctionsplit_into_cols
(items: List[V], npartitions: int)
smallpond/common.py:74
↓ 2 callersMethodstart_execution
Start a scheduler and `num_executors` executors to execute `plan`. When this function returns, the execution is mostly still running.
tests/test_fabric.py:137
↓ 2 callersMethodtake
Take `num_rows` rows from the range. NOTE: this function modifies the current row range.
smallpond/io/arrow.py:53
↓ 2 callersFunctionurls_sort_benchmark
( input_paths: List[str], num_data_partitions: int, num_hash_partitions: int, engine_type="duc
benchmarks/urls_sort_benchmark.py:17
↓ 2 callersMethodvisit
(self, node: Node, depth: int = 0)
smallpond/logical/optimizer.py:16
↓ 2 callersFunctionwrite_to_file
(fout)
smallpond/io/filesystem.py:89
↓ 1 callersMethod__init__
( self, ctx: RuntimeContext, id: str, wq: WorkQueue, cq: WorkQueue, init_epoch=0 )
smallpond/execution/scheduler.py:57
↓ 1 callersMethod__init__
( self, exec_plan: ExecutionPlan, max_retry_count: int = DEFAULT_MAX_RETRY_COUNT,
smallpond/execution/scheduler.py:381
↓ 1 callersMethod__init__
( self, data_root: Optional[str] = None, python_venv: Optional[str] = None, ta
smallpond/execution/manager.py:53
↓ 1 callersMethod__init__
( self, *args, sleep_secs: float, fail_first_try: bool, **kwargs )
tests/test_scheduler.py:24
↓ 1 callersMethod__repr__
(self)
smallpond/execution/scheduler.py:78
↓ 1 callersMethod__str__
(self)
smallpond/logical/dataset.py:99
↓ 1 callersMethod_all_tasks_finished
Check if all tasks are finished.
smallpond/dataframe.py:191
← previousnext →101–200 of 992, ranked by callers