Get the pad token safely. It always returns a pad token, which is the token closest to pad if not presented in the vocab. Args:

Get the pad token safely. It always returns a pad token, which is the pad token or the first token if pad does not present in the vocab.

Get the idx to the pad token safely. It always returns an index, which corresponds to the pad token or the first token if pad does not present

Get the unk token safely. It always returns a unk token, which is the token closest to unk if not presented in the vocab. Args:

Get the unk token safely. It always returns a unk token, which is the unk token or the first token if unk does not presented in the vocab.

Save vocabularies to a directory. Args: save_dir: The directory to save vocabularies. filename: The name for vocabul

Semantic textual similarity deals with determining how similar two pieces of texts are. @param textA The first text. @param textB The second text. @r

Semantic textual similarity deals with determining how similar two pieces of texts are. Args: text: A pair or pairs of text.

Sentiment analysis is the task of classifying the polarity of a given text. For instance, a text-based tweet can be categorized into either "positive"

r""" Sentiment analysis is the task of classifying the polarity of a given text. For instance, a text-based tweet can be categorized i

A function to split apostrophe contractions at the end of alphanumeric (and hyphenated) tokens. Takes the output of any of the tagger functi

Split document-based samples into sentence-based samples for evaluation. Args: example: Returns:

A function to split possessive markers at the end of alphanumeric (and hyphenated) tokens. Takes the output of any of the tagger functions a

Whether the training procedure should perform an update. Args: gradient_accumulation: Number of batches per update. Ret

Log a summary of vocabs using a given logger. Args: logger: The logger to use.

The symbol tagger extends the :func:`space_tokenizer` by separating alphanumerics. Separates alphanumeric Unicode character sequences in alr