tokenize_sentences function

Sentence tokenizer

Get sentence tokens from text

tokenize_sentences(input, EOS = ".?!:;")

Arguments

• input: a character vector.
• EOS: a length one character vector listing all (single character) end-of-sentence tokens.

Returns

a character vector, each entry of which corresponds to a single sentence.

Examples

tokenize_sentences("Hi there! I'm using `sbo`.")

Author(s)

Valerio Gherardi