crawl_delays function

Retrieve all agent crawl delay values in a robxp robots.txt object

crawl_delays(obj)

Arguments

• obj: robxp object

Returns

data frame of agents and their crawl delays

Note

-1 will be returned for any listed agent without a crawl delay setting

Examples

gh <- paste0(readLines(system.file("extdata", "github-robots.txt",
             package="spiderbar")), collapse="\n")
gh_rt <- robxp(gh)
crawl_delays(gh_rt)

imdb <- paste0(readLines(system.file("extdata", "imdb-robots.txt",
               package="spiderbar")), collapse="\n")
imdb_rt <- robxp(imdb)
crawl_delays(imdb_rt)