map can work with multiple kinds of inputs and outputs Input Types: 1. single value 2. (key, value) : Most common format for key value pair 3. (key, v

Close implements io.Closer, and closes the current logfile.

hash data or by data key, return a new dataset This is devided into 2 steps: 1. Each record is sharded to a local shard 2. The destination shard will

assume nothing about these two dataset

f(V, V) bool : less than function New Dataset contains K,V

TextFile returns a new Dataset which reads the text file fname line by line, and distributes them evenly among multiple shards.

f(A)bool

Slice accepts a slice as the input and sends the received values to tasks via Channel().

Source returns a new Dataset which evenly distributes the data items produced by f among multiple shards. f must be a function defined in the form fun

openNew opens a new log file for writing, moving any old log file out of the way. This methods assumes the file has already been closed.

retChan should be a buffered channel

collectOutput collects the output of d and returns them as a slice of the type specified in d.Type. Intends to be used in tests. The implementation d

dir returns the directory for the current filename.

max returns the maximum size in bytes of log files before rolling.

create a channel to aggregate values of the same key automatically close original sorted channel

ReduceByKey runs on a dataset with (K, V) pairs, returns a dataset with (K, V) pairs, where values for the same key are aggreated by function f. Funct

Starts a http server to serve '/' with the input handler. Returns the listener or any error.

rotate closes the current file, moves it aside with a timestamp in the name, (if it exists), opens a new file with the original filename, and then run

the task should run on the destination dataset shard

the tasks should run on the source dataset shard

big endian

Channel returns a new Dataset which has the input channel as the input and sends the received values to tasks.