Access to Large Language Model Predictions
Preloads a causal language model
Get the log probability of each token in a sentence (or group of sente...
Returns the configuration of a causal model
Get a list of matrices with the log probabilities of possible words gi...
Get the log probability of each element of a vector of words (or phras...
Generate next tokens after a context and their predictability using a ...
Get the possible next tokens and their log probabilities for its previ...
Generate a list of predictability matrices using a causal transformer ...
Compute predictability using a causal transformer model
Install the Python packages needed for pangoling
Check if the required Python dependencies for pangoling are installe...
Returns the configuration of a masked model
Get the log probability of a target word (or phrase) given a left and ...
Preloads a masked language model
Get the predictability of a target word (or phrase) given a left and r...
Get the possible tokens and their log probabilities for each mask in a...
Get the possible tokens and their log probabilities for each mask in a...
The number of tokens in a string or vector of strings
Defunct functions in package pangoling.
pangoling: Access to Large Language Model Predictions
Calculates perplexity
Set cache folder for HuggingFace transformers
Tokenize an input
Returns the vocabulary of a model
Provides access to word predictability estimates using large language models (LLMs) based on 'transformer' architectures via integration with the 'Hugging Face' ecosystem <https://huggingface.co/>. The package interfaces with pre-trained neural networks and supports both causal/auto-regressive LLMs (e.g., 'GPT-2') and masked/bidirectional LLMs (e.g., 'BERT') to compute the probability of words, phrases, or tokens given their linguistic context. For details on GPT-2 and causal models, see Radford et al. (2019) <https://storage.prod.researchhub.com/uploads/papers/2020/06/01/language-models.pdf>, for details on BERT and masked models, see Devlin et al. (2019) <doi:10.48550/arXiv.1810.04805>. By enabling a straightforward estimation of word predictability, the package facilitates research in psycholinguistics, computational linguistics, and natural language processing (NLP).
Useful links