Text Tokenization using Byte Pair Encoding and Unigram Modelling
Tokenise and embed text alongside a Sentencepiece and Word2vec model
Build a BPEembed model containing a Sentencepiece and Word2vec model
Encode and Decode alongside a BPEembed model
Read a word2vec embedding file
Construct a Sentencepiece model
Decode encoded sequences back to text
Download a Sentencepiece model
Tokenise text alongside a Sentencepiece model
Load a Sentencepiece model
Remove prefixed underscore
Wordpiece encoding
Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling. Wraps the 'sentencepiece' library <https://github.com/google/sentencepiece> which provides a language independent tokenizer to split text in words and smaller subword units. The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) <doi:10.18653/v1/D18-2012>. Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec', as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) <http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf>.