Remove the spans attached to tokens and return a new document. Args: tok: The key to tokens. Returns:

Tokenize a string. Modified in order to handle sentences with recategorization pointers

Tokenize a string. Modified in order to handle sentences with recategorization pointers

类似 -3 -2 -1 0 1 2 -30 -20 -10 00 10 20 -300 -200 -100 000 100 200 转换为 0 -10 -200

Add additional bytes content as read from the read_position. Args: content(bytes): data to be added to buffer working BufferWorkSpa

Given labels and a constraint type, returns the allowed transitions. It will additionally include transitions for the start and end states, w

Args: input: B x * x ... x * index: B x M dim: (Default value = 1) Returns:

r""" Conducts binarization over the tree. First, the tree is transformed to satisfy `Chomsky Normal Form (CNF)`_. Here we call :meth:`~tr

Build a ``Sampler`` given statistics of samples and other arguments. Args: lengths: The lengths of samples. shuffle:

Implement this method to build an optimizer. Args: **kwargs: The subclass decides the method signature.

Build a transformer tokenizer for this task. Args: tokenizer: A tokenizer which is shared but can be adjusted to provide per-task

r""" Builds a constituency tree from the sequence. The sequence is generated in pre-order. During building the tree, the sequence is de-binari