MCPcopy Create free account

hub / github.com/huggingface/transformers / functions

Functions4,119 in github.com/huggingface/transformers

↓ 1 callersFunctionmain
()
examples/question-answering/run_tf_squad.py:116
↓ 1 callersFunctionmake_eval_dict
(exact_scores, f1_scores, qid_list=None)
src/transformers/data/metrics/squad_metrics.py:107
↓ 1 callersMethodmake_example
(self, idx)
examples/longform-qa/eli5_utils.py:99
↓ 1 callersMethodmake_example
(self, idx)
examples/longform-qa/eli5_utils.py:347
↓ 1 callersFunctionmake_padding_mask
True for pad tokens
src/transformers/modeling_bart.py:186
↓ 1 callersFunctionmake_qa_s2s_batch
(qa_list, tokenizer, max_len=64, max_a_len=360, device="cuda:0")
examples/longform-qa/eli5_utils.py:375
↓ 1 callersFunctionmake_qa_s2s_model
(model_name="facebook/bart-large", from_file=None, device="cuda:0")
examples/longform-qa/eli5_utils.py:366
↓ 1 callersFunctionmake_registry
(repo_path="Opus-MT-train/models")
src/transformers/convert_marian_to_pytorch.py:189
↓ 1 callersMethodmake_sortish_sampler
(self, batch_size)
examples/seq2seq/utils.py:145
↓ 1 callersFunctionmake_support
(question, source="wiki40b", method="dense", n_results=10)
examples/longform-qa/eli5_app.py:88
↓ 1 callersMethodmap_batch_fn
(self, fn)
examples/seq2seq/bertabs/modeling_bertabs.py:558
↓ 1 callersFunctionmask_heads
This method shows how to mask head (set some heads to zero), to test the effect on the network, based on the head importance scores, as descr
examples/bertology/run_bertology.py:152
↓ 1 callersMethodmask_tokens
Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.
src/transformers/data/data_collator.py:102
↓ 1 callersFunctionmerge_eval
(main_eval, new_eval, prefix)
src/transformers/data/metrics/squad_metrics.py:128
↓ 1 callersMethodmerge_heads
(self, x)
src/transformers/modeling_tf_openai.py:134
↓ 1 callersMethodmerge_heads
(self, x)
src/transformers/modeling_tf_gpt2.py:127
↓ 1 callersMethodmerge_heads
(self, x)
src/transformers/modeling_gpt2.py:170
↓ 1 callersMethodmerge_heads
(self, x)
src/transformers/modeling_openai.py:188
↓ 1 callersMethodmodel
(self)
tests/test_modeling_marian.py:84
↓ 1 callersMethodmoses_punct_norm
(self, text, lang)
src/transformers/tokenization_xlm.py:656
↓ 1 callersMethodmove_added_token
Moves an added token to a specific position in the vocab. This method should be used when resizing an embedding layer other than the
src/transformers/tokenization_transfo_xl.py:275
↓ 1 callersMethodnum_special_tokens_to_add
Returns the number of added tokens when encoding a sequence with special tokens. Note: This encodes inputs and checks th
src/transformers/tokenization_utils.py:237
↓ 1 callersMethodoptimize
Normalization on the loss (gradient accumulation or distributed training), followed by backward pass on the loss, possibly followed b
examples/distillation/distiller.py:468
↓ 1 callersMethodoptimizer_step
(self, epoch, batch_idx, optimizer, optimizer_idx, second_order_closure=None)
examples/lightning_base.py:119
↓ 1 callersFunctionpad_sequences
(sequences)
examples/text-generation/pplm/run_pplm_discrim_train.py:102
↓ 1 callersFunctionparseGithubButtons
* modified to run programmatically
docs/source/_static/js/custom.js:246
↓ 1 callersFunctionparse_args
Helper function parsing the command line options @retval ArgumentParser
examples/xla_spawn.py:22
↓ 1 callersFunctionparse_flag_from_env
(key, default=False)
tests/utils.py:13
↓ 1 callersFunctionparse_int_from_env
(key, default=None)
tests/utils.py:29
↓ 1 callersFunctionpassage_generator
()
examples/longform-qa/eli5_utils.py:46
↓ 1 callersFunctionpearson_and_spearman
(preds, labels)
src/transformers/data/metrics/__init__.py:44
↓ 1 callersFunctionperturb_past
( past, model, last, unpert_past=None, unpert_logits=None, accumulated_hidden=None,
examples/text-generation/pplm/run_pplm.py:94
↓ 1 callersFunctionpickle_load
pickle.load(path)
examples/seq2seq/utils.py:182
↓ 1 callersFunctionplatformToggle
()
docs/source/_static/js/custom.js:156
↓ 1 callersMethodplot
(self)
examples/benchmarking/plot_csv_file.py:87
↓ 1 callersFunctionpoint_wise_feed_forward_network
(d_model_size, dff)
src/transformers/modeling_ctrl.py:166
↓ 1 callersFunctionpoint_wise_feed_forward_network
(d_model_size, dff, name="")
src/transformers/modeling_tf_ctrl.py:143
↓ 1 callersMethodpositional_embedding
(pos_seq, inv_freq, bsz=None)
src/transformers/modeling_xlnet.py:704
↓ 1 callersMethodpositional_embedding
(pos_seq, inv_freq, bsz=None)
src/transformers/modeling_tf_xlnet.py:456
↓ 1 callersFunctionpositional_encoding
(position, d_model_size, dtype)
src/transformers/modeling_ctrl.py:46
↓ 1 callersFunctionpositional_encoding
(position, d_model_size)
src/transformers/modeling_tf_ctrl.py:51
↓ 1 callersMethodpost_attention
Post-attention processing.
src/transformers/modeling_xlnet.py:307
↓ 1 callersMethodpost_attention
Post-attention processing.
src/transformers/modeling_tf_xlnet.py:189
↓ 1 callersMethodpre_init
(self, hparams)
examples/seq2seq/distillation.py:74
↓ 1 callersFunctionpre_process_datasets
Pre-process datasets containing lists of tuples(story, 1st continuation, 2nd continuation, label) To Transformer inputs of shape (n_batch, n
examples/contrib/run_openai_gpt.py:73
↓ 1 callersFunctionpredict
(input_sentence, model, classes, cached=False, device="cpu")
examples/text-generation/pplm/run_pplm_discrim_train.py:185
↓ 1 callersMethodpredict
Run prediction and return predictions and potential metrics. Depending on the dataset and your use case, your test dataset may contai
src/transformers/trainer_tf.py:479
↓ 1 callersMethodprepare_batch_clm
Prepare the batch: from the token_ids and the lenghts, compute the attention mask and the labels for CLM. Input: ------
examples/distillation/distiller.py:254
↓ 1 callersMethodprepare_batch_mlm
Prepare the batch: from the token_ids and the lenghts, compute the attention mask and the masked label for MLM. Input: -----
examples/distillation/distiller.py:189
↓ 1 callersMethodprepare_config_and_inputs_for_common
(self)
tests/test_modeling_transfo_xl.py:156
↓ 1 callersMethodprepare_config_and_inputs_for_question_answering
(self)
tests/test_modeling_longformer.py:249
↓ 1 callersMethodprepare_for_tokenization
Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return kwargs as well.
src/transformers/tokenization_utils.py:726
↓ 1 callersMethodprepare_kwargs
(self)
tests/test_modeling_reformer.py:479
↓ 1 callersMethodprepare_kwargs
(self)
tests/test_modeling_reformer.py:533
↓ 1 callersMethodpreprocess_text
(self, inputs)
src/transformers/tokenization_albert.py:184
↓ 1 callersMethodpreprocess_text
(self, text)
src/transformers/tokenization_flaubert.py:103
↓ 1 callersMethodpreprocess_text
(self, inputs)
src/transformers/tokenization_xlnet.py:187
↓ 1 callersMethodprint_memory_trace_statistics
(self, summary: MemorySummary)
src/transformers/benchmark/benchmark_utils.py:816
↓ 1 callersMethodprint_results
(self, result_dict, type_label)
src/transformers/benchmark/benchmark_utils.py:794
↓ 1 callersMethodprint_statistics
Print some statistics on the corpus. Only the master process.
examples/distillation/lm_seqs_dataset.py:129
↓ 1 callersFunctionprune_heads
This method shows how to prune head (remove heads weights) based on the head importance scores as described in Michel et al. (http://arxiv.or
examples/bertology/run_bertology.py:203
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_distilbert.py:119
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_ctrl.py:102
↓ 1 callersMethodprune_heads
Prunes heads of the base model. Arguments: heads_to_prune: dict with keys being selected layer indices (`int`) and asso
src/transformers/modeling_utils.py:458
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_bert.py:287
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_albert.py:200
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_mobilebert.py:291
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_tf_xlm.py:134
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_t5.py:217
↓ 1 callersMethodprune_heads
(self, heads)
src/transformers/modeling_openai.py:148
↓ 1 callersFunctionqa_s2s_generate
( question_doc, qa_s2s_model, qa_s2s_tokenizer, num_answers=1, num_beams=None, min_len
examples/longform-qa/eli5_utils.py:494
↓ 1 callersFunctionquery_es_index
(question, es_client, index_name="english_wiki_kilt_snippets_100w", n_results=10, min_length=20)
examples/longform-qa/eli5_utils.py:57
↓ 1 callersFunctionquery_qa_dense_index
( question, qa_embedder, tokenizer, wiki_passages, wiki_index, n_results=10, min_length=20, device="cuda:0
examples/longform-qa/eli5_utils.py:595
↓ 1 callersFunctionrandom_input_ids
(batch_size: int, sequence_length: int, vocab_size: int)
src/transformers/benchmark/benchmark_tf.py:72
↓ 1 callersFunctionread_squad_examples
Read a SQuAD json file into a list of SquadExample.
templates/adding_a_new_example_script/utils_xxx.py:111
↓ 1 callersFunctionread_swag_examples
(input_file, is_training=True)
examples/contrib/run_swag.py:92
↓ 1 callersMethodregister_subcommand
(parser: ArgumentParser)
src/transformers/commands/run.py:50
↓ 1 callersFunctionregularization
(model: nn.Module, mode: str)
examples/movement-pruning/masked_run_glue.py:93
↓ 1 callersFunctionregularization
(model: nn.Module, mode: str)
examples/movement-pruning/masked_run_squad.py:97
↓ 1 callersMethodrel_attn_core
Core relative positional attention operations.
src/transformers/modeling_xlnet.py:255
↓ 1 callersMethodrel_attn_core
Core relative positional attention operations.
src/transformers/modeling_tf_xlnet.py:144
↓ 1 callersMethodrel_shift
perform relative shift to form the relative attention score.
src/transformers/modeling_tf_xlnet.py:132
↓ 1 callersMethodrel_shift_bnij
(x, klen=-1)
src/transformers/modeling_xlnet.py:241
↓ 1 callersMethodrelative_positional_encoding
(self, qlen, klen, bsz=None)
src/transformers/modeling_xlnet.py:714
↓ 1 callersMethodrelative_positional_encoding
create relative positional encoding.
src/transformers/modeling_tf_xlnet.py:466
↓ 1 callersFunctionremove_articles
(text)
src/transformers/data/metrics/squad_metrics.py:27
↓ 1 callersMethodremove_empty_sequences
Too short sequences are simply removed. This could be tunedd.
examples/distillation/lm_seqs_dataset.py:102
↓ 1 callersMethodremove_language_code
Remove language codes like <<fr>> before sentencepiece
src/transformers/tokenization_marian.py:99
↓ 1 callersMethodremove_long_sequences
Sequences that are too long are splitted by chunk of max_model_input_size.
examples/distillation/lm_seqs_dataset.py:62
↓ 1 callersFunctionremove_non_printing_char
Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/remove-non-printing-char.perl
src/transformers/tokenization_xlm.py:503
↓ 1 callersFunctionremove_punc
(text)
src/transformers/data/metrics/squad_metrics.py:34
↓ 1 callersMethodremove_unknown_sequences
Remove sequences with a (too) high level of unknown tokens.
examples/distillation/lm_seqs_dataset.py:113
↓ 1 callersFunctionrename_key
(dct, old, new)
src/transformers/convert_bart_original_pytorch_checkpoint_to_pytorch.py:68
↓ 1 callersFunctionreplace_unicode_punct
Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/replace-unicode-punctuation.perl
src/transformers/tokenization_xlm.py:460
↓ 1 callersMethodreset_length
(self, tgt_len, ext_len, mem_len)
src/transformers/modeling_tf_transfo_xl.py:482
↓ 1 callersMethodreset_memory_hooks_state
(self)
src/transformers/modeling_utils.py:119
↓ 1 callersMethodreset_stats
(self)
examples/bert-loses-patience/pabee/modeling_pabee_bert.py:86
↓ 1 callersFunctionromanian_preprocessing
Sennrich's WMT16 scripts for Romanian preprocessing, used by model `xlm-mlm-enro-1024`
src/transformers/tokenization_xlm.py:516
↓ 1 callersFunctionrun_pplm_example
( pretrained_model="gpt2-medium", cond_text="", uncond=False, num_samples=1, bag_of_words=
examples/text-generation/pplm/run_pplm.py:574
↓ 1 callersMethodrun_tf
(self)
src/transformers/commands/train.py:132
← previousnext →1,201–1,300 of 4,119, ranked by callers