robxp function

Parse a robots.txt file & create a robxp object

This function takes in a single element character vector and parses it into a robxp object.

robxp(x)

Arguments

• x: either an atomic character vector containing a complete robots.txt`` file _or_ a length >1 character vector that will concatenated into a single string _or_ a connection` object that will be passed to [readLines()], the result of which will be concatenated into a single string and parsed and the connection will be closed.

Returns

a classed object holding an external pointer to parsed robots.txt data

Examples

imdb <- paste0(readLines(system.file("extdata", "imdb-robots.txt",
               package="spiderbar")), collapse="\n")
rt <- robxp(imdb)