LM wrapper that returns cached results if they exist, and uses the underlying LM if not. :param lm: LM Underlying LM :par

SAT Analog Questions is not publicly available. You must request the data by emailing Peter Turney and then download it to a local di

StoryCloze is not publicly available. You must download the data by following https://cs.rochester.edu/nlp/rocstories/ and pass the f

BuilderConfig for Wikitext Args: data_url: `string`, url to the dataset (word or raw level) **kwargs: keyword arguments fo

BuilderConfig for Hendrycks ETHICS. Args: prefix: *string*, prefix to add to the dataset name for path location. features: *l

BuilderConfig for GPT3 Arithmetic dataset. Args: url: *string*, the url to the specific subset of the GPT3 Arithmetic dataset.

:param engine: str TextSynth API engine (e.g. `gptj_6B`) :param truncate: bool Truncate input if too long (if

:param engine: str OpenAI API engine (e.g. davinci) :param truncate: bool Truncate input if too long (if Fal

binary_cmd: the command to start the inference binary

LM wrapper that returns cached results if they exist, and uses the underlying LM if not. :param lm: LM Underlying LM :par

SAT Analog Questions is not publicly available. You must request the data by emailing Peter Turney and then download it to a local di

StoryCloze is not publicly available. You must download the data by following https://cs.rochester.edu/nlp/rocstories/ and pass the f

BuilderConfig for Wikitext Args: data_url: `string`, url to the dataset (word or raw level) **kwargs: keyword arguments fo

BuilderConfig for Hendrycks ETHICS. Args: prefix: *string*, prefix to add to the dataset name for path location. features: *l

BuilderConfig for GPT3 Arithmetic dataset. Args: url: *string*, the url to the specific subset of the GPT3 Arithmetic dataset.

:param engine: str TextSynth API engine (e.g. `gptj_6B`) :param truncate: bool Truncate input if too long (if

:param engine: str OpenAI API engine (e.g. davinci) :param truncate: bool Truncate input if too long (if Fal

Calculate perplexity using the same method as seen in llama.cpp. Parameters ---------- model : AutoModelForCausalLM