Append a pipe to the tail of this pipeline. Args: component: A callable function. input_key: The input key i

Split dataset into subsets. Args: *ratios: The ratios for each subset. They can be any type of numbers which will be normalized.

Tries to add a token into a vocab and returns its id. If it has already been there, its id will be returned and the vocab won't be updated. I

Fetch real (local) path for a resource (model, corpus, whatever) to ``save_dir``. Args: path: A local path (which will returned as is) or a

Update the vocab with these tokens by adding them to vocab one by one. Args: tokens (Iterable[str]): A list of tokens.

cd to project root, so models are saved in the root folder

Evaluate test set. Args: tst_data: Test set, which is usually a file path. save_dir: The directory to save evaluation

Move this component to devices. Args: devices: Target devices. logger: Logger for printing progress report, as copyin

Build the vocabulary from training file Parameters ---------- trn_path : path to training set kwargs

Put names and corresponding :class:`hanlp.common.vocab.Vocab` instances into self. Args: **kwargs: Names and corresponding :class

r'''Returns a diagonal stripe of the tensor. Args: x: Tensor n: int w: int offset: tuple (Default value = (0) dim:

Convert all tokens to lower case. Returns: Itself.

r""" Squeeze the dimension of each field into one. It's intended to convert a nested document like ``[[sent_i]]`` to ``[sent_i]``. Whe

Args: index: The index of the new pipe. input_key: The input key indicating which fields will be inputted to the pip

Append a transform to its list of transforms. Args: transform: A new transform to be appended. Returns: Itse

Merging the provided dict with other kvs Args: d: kwargs: d: dict: overwrite: (Default value = False) inplace:

Gets entities from sequence. Args: seq(list): sequence of labels. suffix: (Default value = False) Returns: list: list of

Get the idx of a token. If it's not there, it will be added to the vocab when the vocab is locked otherwise the id of UNK will be returned.

Like the :meth:`str.split`, this method splits a piece of text into chunks by taking the keys in this dictionary as delimiters. It performs lo

Print a pretty text representation which visualizes linguistic structures. Args: tok: Token key. lem: Lemma

Purges all cache. If cache is not enabled, this method enables it.

get the full class path of object Args: obj: return: Returns:

Lock this vocab up so that it won't accept new tokens. Returns: Itself.

Get or print a summary of this vocab. Args: verbose: ``True`` to print the summary to stdout. Returns: Summa

Args: data: Samples to be measured or path to dataset during training time. dataset: During training time, use this

Convert this vocab to a dict so that it can be json serialized. Returns: A dict.

Build a module for this embedding. Args: **kwargs: Containing vocabs, training etc. Not finalized for now. Returns:

Perform the actual padding for a given data. Args: data: Data to be padded. pad: Padding value. dtype: Da

Shuffle this dataset inplace.

The first device this component lives on.

Transform file to dataset Parameters ---------- filepath gold : bool Whether it's processing gol

See Also https://stackoverflow.com/questions/333995/how-to-detect-that-python-code-is-being-executed-through-the-debugger

Caculate number of training steps. Args: num_batches: Size of dataloader. gradient_accumulation: Number of batches p

Set ``self.unk_token = self.safe_unk_token``. It's useful when the dev/test set contains OOV labels.

convert class path in str format to a type Args: classpath: class path Returns: type

Set of variables in this semantic graph

Transform file to samples Parameters ---------- filepath gold

Transit the state of a Deterministic Finite Automata (DFA) with key. Args: key: A sequence of criterion (tokens or characters) us