MCPcopy Create free account

hub / github.com/huggingface/transformers / functions

Functions4,119 in github.com/huggingface/transformers

Method_tokenize
Take as input a string and return a list of strings (tokens) for words/sub-words
templates/adding_a_new_model/tokenization_xxx.py:132
Method_tokenize
(self, text)
src/transformers/tokenization_xlm_roberta.py:269
Method_tokenize
Tokenize a string.
src/transformers/tokenization_openai.py:175
Method_tokenize
Tokenize a string.
src/transformers/tokenization_albert.py:199
Method_tokenize
Tokenize a string given language code using Moses. Details of tokenization: - [sacremoses](https://github.com/alvations/sacr
src/transformers/tokenization_flaubert.py:113
Method_tokenize
Take as input a string and return a list of strings (tokens) for words/sub-words
src/transformers/tokenization_reformer.py:144
Method_tokenize
Tokenize a string.
src/transformers/tokenization_gpt2.py:222
Method_tokenize
(self, line, add_eos=False, add_double_eos=False)
src/transformers/tokenization_transfo_xl.py:339
Method_tokenize
Tokenize a string.
src/transformers/tokenization_ctrl.py:204
Method_tokenize
Tokenize a string given language code. For Chinese, Japanese and Thai, we use a language specific tokenizerself. Otherwise, we use Moses.
src/transformers/tokenization_xlm.py:749
Method_tokenize
Take as input a string and return a list of strings (tokens) for words/sub-words
src/transformers/tokenization_t5.py:166
Method_tokenize
Tokenize a string.
src/transformers/tokenization_xlnet.py:202
Method_tokenize
(self, text)
src/transformers/tokenization_bert_japanese.py:153
Method_tokenize
(self, text)
src/transformers/tokenization_bert.py:208
Method_tokenize
(self, text: str)
src/transformers/tokenization_marian.py:105
Method_tokenize
(self, text)
src/transformers/tokenization_camembert.py:227
Method_train_memory
( self, model_name: str, batch_size: int, sequence_length: int )
src/transformers/benchmark/benchmark.py:79
Method_train_memory
( self, model_name: str, batch_size: int, sequence_length: int )
src/transformers/benchmark/benchmark_utils.py:619
Method_train_memory
(self, model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_tf.py:111
Method_train_speed
(self, model_name: str, batch_size: int, sequence_length: int)
src/transformers/benchmark/benchmark.py:75
Method_train_speed
(self, model_name: str, batch_size: int, sequence_length: int)
src/transformers/benchmark/benchmark_utils.py:609
Method_train_speed
(self, model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_tf.py:95
Methodaccumulate
(grad1, grad2)
tests/test_optimization_tf.py:60
Methodaccumulate_on_replica
(gradient)
tests/test_optimization_tf.py:53
Functionadd_code_sample_docstrings
(*docstr, tokenizer_class=None, checkpoint=None)
src/transformers/file_utils.py:417
Functionadd_end_docstrings
(*docstr)
src/transformers/file_utils.py:181
Methodadd_memory_hooks
Add a memory hook before and after each sub-module forward pass to record increase in memory consumption. Increase in memory consumption
src/transformers/modeling_utils.py:110
Methodadd_model_specific_args
(parser, root_dir)
examples/lightning_base.py:176
Functionadd_start_docstrings
(*docstr)
src/transformers/file_utils.py:155
Functionadd_start_docstrings_to_callable
(*docstr)
src/transformers/file_utils.py:163
Methodadditional_special_tokens
All the additional special tokens you may want to use (list of strings). Log an error if used while not having been set.
src/transformers/tokenization_utils_base.py:774
Methodadditional_special_tokens_ids
Ids of all the additional special tokens in the vocabulary (list of integers). Log an error if used while not having been set.
src/transformers/tokenization_utils_base.py:868
Methodadjust_logits_during_generation
(self, logits, cur_len, max_length)
src/transformers/modeling_bart.py:1028
Methodadjust_logits_during_generation
(self, logits, cur_len, max_length)
src/transformers/modeling_marian.py:49
Methodall_special_ids
List the vocabulary indices of the special tokens ('<unk>', '<cls>'...) mapped to class attributes (cls_token, unk_token...).
src/transformers/tokenization_utils_base.py:926
Methodall_special_tokens
List all the special tokens ('<unk>', '<cls>'...) mapped to class attributes Convert tokens of AddedToken type in string. All
src/transformers/tokenization_utils_base.py:902
Methodall_special_tokens_extended
List all the special tokens ('<unk>', '<cls>'...) mapped to class attributes Keep the tokens as AddedToken if they are of this type.
src/transformers/tokenization_utils_base.py:912
Methodapply_grad
()
tests/test_optimization_tf.py:68
Methodapply_on_replica
()
tests/test_optimization_tf.py:56
Methodassert_add_special_tokens
(self, tokenizer_r)
tests/test_tokenization_fast.py:731
Methodassert_batch_padded_input_match
(input_r: dict, input_p: dict, max_length: int)
tests/test_tokenization_fast.py:454
Methodassert_build_inputs_with_special_tokens
(self, tokenizer_r, tokenizer_p)
tests/test_tokenization_fast.py:413
Methodassert_embeded_special_tokens
(self, tokenizer_r, tokenizer_p)
tests/test_tokenization_fast.py:783
Functionassert_not_all_frozen
(model)
examples/seq2seq/utils.py:259
Methodassert_padded_input_match
(input_r: list, input_p: list, max_length: int)
tests/test_tokenization_fast.py:443
Methodassert_save_pretrained
(self, tokenizer_r, tokenizer_p)
tests/test_tokenization_fast.py:650
Methodbackend_tokenizer
(self)
src/transformers/tokenization_utils_fast.py:136
Methodbackward
(ctx, grad_hidden_states)
src/transformers/modeling_reformer.py:1297
Methodbackward
(ctx, gradOutput)
examples/movement-pruning/emmental/modules/binarizer.py:65
Methodbackward_compatible
(self)
src/transformers/modeling_tf_transfo_xl.py:479
Methodbackward_compatible
(self)
src/transformers/modeling_transfo_xl.py:705
Methodbase_model
(self)
src/transformers/modeling_utils.py:306
Functionbatch_query_qa_dense_index
(questions, qa_embedder, tokenizer, wiki_passages, wiki_index, n_results=10)
examples/longform-qa/eli5_utils.py:609
Functionbatch_query_qa_dense_index_nn
(passages, qa_embedder, tokenizer, wiki_passages, wiki_index, n_results=10)
examples/longform-qa/eli5_utils.py:639
Methodbatch_sequences
Do the padding and transform into torch.tensor.
examples/distillation/lm_seqs_dataset.py:144
Methodbeam_update
Need to document this
examples/seq2seq/bertabs/modeling_bertabs.py:546
Methodbig_tokenizer
(self)
tests/test_tokenization_xlm_roberta.py:118
Methodbos_token
Beginning of sentence token (string). Log an error if used while not having been set.
src/transformers/tokenization_utils_base.py:718
Methodbos_token_id
Id of the beginning of sentence token in the vocabulary. Log an error if used while not having been set.
src/transformers/tokenization_utils_base.py:814
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_electra.py:492
Methodbuild
Build shared word embedding layer
src/transformers/modeling_tf_albert.py:89
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_transfo_xl.py:158
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_transfo_xl.py:350
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_transfo_xl.py:460
Methodbuild
Build shared word embedding layer
src/transformers/modeling_tf_mobilebert.py:120
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_mobilebert.py:597
Methodbuild
Build shared token embedding layer Shared weights logic adapted from https://github.com/tensorflow/models/blob/a009f4fb9d2fc4949e3
src/transformers/modeling_tf_utils.py:1626
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_bert.py:465
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_distilbert.py:616
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_roberta.py:251
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_xlnet.py:98
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_xlnet.py:402
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_transfo_xl_utilities.py:45
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_xlm.py:663
Methodbuild
Build shared word embedding layer
src/transformers/modeling_tf_t5.py:66
Methodbuild
(self, input_shape)
src/transformers/modeling_tf_ctrl.py:530
Methodbuild_corpus
(self, path, dataset)
src/transformers/tokenization_transfo_xl.py:715
Methodbuild_inputs_with_special_tokens
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens
src/transformers/tokenization_xlm_roberta.py:177
Methodbuild_inputs_with_special_tokens
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens
src/transformers/tokenization_roberta.py:165
Methodbuild_inputs_with_special_tokens
(self, token_ids_0, token_ids_1=None)
src/transformers/tokenization_roberta.py:344
Methodbuild_inputs_with_special_tokens
Build model inputs from a sequence by appending eos_token_id.
src/transformers/tokenization_bart.py:119
Methodbuild_inputs_with_special_tokens
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens
src/transformers/tokenization_xlnet.py:239
Methodbuild_inputs_with_special_tokens
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens
src/transformers/tokenization_bert.py:235
Methodbuild_inputs_with_special_tokens
(self, token_ids_0, token_ids_1=None)
src/transformers/tokenization_bert.py:635
Methodbuild_inputs_with_special_tokens
Build model inputs from a sequence by appending eos_token_id.
src/transformers/tokenization_marian.py:118
Methodbuild_inputs_with_special_tokens
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens
src/transformers/tokenization_camembert.py:142
Functionc
()
docs/source/_static/js/custom.js:246
Functioncached_collate_fn
(data)
examples/text-generation/pplm/run_pplm_discrim_train.py:123
Methodcalc_generative_metrics
(self, preds, target)
examples/seq2seq/finetune.py:288
Methodcall
(self, inputs, training=False)
templates/adding_a_new_model/modeling_tf_xxx.py:77
Methodcall
( self, inputs, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, training=
templates/adding_a_new_model/modeling_tf_xxx.py:102
Methodcall
(self, inputs, **kwargs)
templates/adding_a_new_model/modeling_tf_xxx.py:312
Methodcall
(self, inputs, **kwargs)
templates/adding_a_new_model/modeling_tf_xxx.py:355
Methodcall
(self, inputs, **kwargs)
templates/adding_a_new_model/modeling_tf_xxx.py:408
Methodcall
(self, inputs, **kwargs)
templates/adding_a_new_model/modeling_tf_xxx.py:463
Methodcall
(self, inputs, **kwargs)
templates/adding_a_new_model/modeling_tf_xxx.py:519
Methodcall
Get token embeddings of inputs. Args: inputs: list of three int64 tensors with shape [batch_size, length]: (input_ids, position_id
src/transformers/modeling_tf_electra.py:74
Methodcall
(self, discriminator_hidden_states, training=False)
src/transformers/modeling_tf_electra.py:145
Methodcall
(self, generator_hidden_states, training=False)
src/transformers/modeling_tf_electra.py:160
Methodcall
( self, inputs, attention_mask=None, token_type_ids=None, position_ids
src/transformers/modeling_tf_electra.py:235
← previousnext →2,201–2,300 of 4,119, ranked by callers