MCPcopy Create free account
hub / github.com/google-research/language / encode_map_fn

Method encode_map_fn

language/serene/preprocessing.py:498–538  ·  view source on GitHub ↗

Read example from tfds and constructed named tensor inputs.

(example)

Source from the content-addressed store, hash-verified

496 evidence_segment_ids, claim_label, evidence_label, metadata)
497
498 def encode_map_fn(example):
499 """Read example from tfds and constructed named tensor inputs."""
500 tensors = tf.py_function(
501 encode,
502 inp=[
503 example['claim_text'], example['evidence_text'],
504 example['claim_label'], example['evidence_label'],
505 example['metadata'], example['wikipedia_url'],
506 example['sentence_id']
507 ],
508 Tout=(
509 tf.string,
510 tf.int32,
511 tf.int32,
512 tf.int32,
513 tf.string,
514 tf.int32,
515 tf.int32,
516 tf.int32,
517 tf.int64,
518 tf.int64,
519 tf.string,
520 ))
521 # Keras expects (x, y), where x can be a dictionary.
522 x = {
523 'claim_text_tokens': tensors[0],
524 'claim_text_word_ids': tensors[1],
525 'claim_text_mask': tensors[2],
526 'claim_text_segment_ids': tensors[3],
527 'evidence_text_tokens': tensors[4],
528 'evidence_text_word_ids': tensors[5],
529 'evidence_text_mask': tensors[6],
530 'evidence_text_segment_ids': tensors[7],
531 'claim_label': tensors[8],
532 'evidence_label': tensors[9],
533 'metadata': tensors[10]
534 }
535 # Order matters, must match training.py.
536 # [evidence_label, claim_label]
537 y = {'claim_classification': tensors[8], 'evidence_matching': tensors[9]}
538 return x, y
539
540 return encode_map_fn
541

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected