A word2vec style embedding module which maps a token to its embedding through looking up a pre-defined table. Args: field: The fi

Toy example of Word2VecEmbedding. It simply returns the embedding of a given word Args: **kwargs:

A contextualized word embedding module. Args: field: The field to work on. Usually some token fields. transformer: A

Args: field: The character field in samples this encoder will work on. embed: An ``Embedding`` object or the feature

Character level RNN embedding module builder. Args: field: The field in samples this encoder will work on. embed: An

A ``nn.ModuleList`` to bundle several embeddings modules. Args: *modules: Embedding layers. dropout: Dropout applied

An embedding builder to bundle several embedding builders. Args: *embeddings_: A list of embedding builders. embeddin

A pre-trained transformer encoder. Args: transformer: A ``PreTrainedModel`` or an identifier of a ``PreTrainedModel``.

Args: in_features: num_heads: dropout: r_w_bias: n_head x head_dim or None r_r_bi

Initialize the CrfDecodeForwardRnnCell. Args: transition_params: A [num_tags, num_tags] matrix of binary potentials. Th

An object which can be transformed with a list of functions. It is the final result of an object being passed through a list of functions, whi

A :class:`~torch.utils.data.Dataset` which can be applied with a list of transform functions. Args: data: The local or remote pat

A dataloader commonly used for NLP tasks. It offers the following convenience. - Bachify each field of samples into a :class:`~torch.Tensor`

A dataloader wrapper which speeds up bachifying using multi-processing. It works best for dataloaders of which the bachify takes very long ti

A bucketing based sampler which groups samples into buckets then creates batches from each bucket. Args: buckets: A dict of which

A bucket sampler which groups samples using KMeans on their lengths. Args: lengths: Lengths of each sample, usually measured by n

A sampler which sorts samples according to their lengths. It takes a continuous chunk of sorted samples to make a batch. The effective batch s

Builds a :class:`~hanlp.common.dataset.SortingSampler`. Args: batch_max_tokens: Maximum tokens per batch. use_effecti

Builds a :class:`~hanlp.common.dataset.KMeansSampler`. Args: batch_max_tokens: Maximum tokens per batch. batch_size:

A dict holding :class:`hanlp.common.vocab.Vocab` instances. When used as a transform, it transforms the field corresponding to each :class:`ha

Mask out all punctuations (set mask of punctuations to False) Args: src: dst: Returns:

This base class helps sub-classes to capture their arguments passed to ``__init__``, and also their types so that they can be deserialized fro

A history of training context. It records how many steps have passed and provides methods to decide whether an update should be performed, an

The base class for all components using PyTorch as backend. It provides common workflows of building vocabs, datasets, dataloaders and models.

A dataset for tagging tokenization tasks. Args: data: The local or remote path to a dataset, or a list of samples where each samp

A dataset for ``.jsonlines`` format NER corpora. Args: data: The local or remote path to a dataset, or a list of samples where ea

Args: data: The local or remote path to a dataset, or a list of samples where each sample is a dict. transform: Pred

General class for CoNLL style dependency parsing datasets. Args: data: The local or remote path to a dataset, or a list of sample

Dataset for sentence boundary detection (eos). Args: data: The local or remote path to a dataset, or a list of samples where each

A transformer tokenizer for token-level tasks. It honors the boundary of tokens and tokenize each token into several subtokens then merge them

CoNLL-U format template, see https://universaldependencies.org/format.html Args: id (Union[int, str]): Token cou

A list of :class:`~hanlp_common.conll.CoNLLWord` or :class:`~hanlp_common.conll.CoNLLUWord`. It is a sub-class of :class:`list` and i

r"""A dict structure holding parsed annotations. A document is a subclass of ``dict`` and it supports every interface of ``dict``\. Additional