preprocess_tokens function

Preprocess tokens in a character vector