Rcrawler0.1.9-1 package

Web Crawler and Scraper

browser_path

Return browser (webdriver) location path

ContentScraper

ContentScraper

Drv_fetchpage

Fetch page using web driver/Session

Getencoding

Getencoding

install_browser

Install PhantomJS webdriver

LinkExtractor

LinkExtractor

LinkNormalization

Link Normalization

Linkparameters

Get the list of parameters and values from an URL

Linkparamsfilter

Link parameters filter

ListProjects

ListProjects

LoadHTMLFiles

LoadHTMLFiles @rdname LoadHTMLFiles

LoginSession

Open a logged in Session

Rcrawler

Rcrawler

RobotParser

RobotParser fetch and parse robots.txt

run_browser

Start up web driver process on localhost, with a random port

stop_browser

Stop web driver process and Remove its Object

Download source packageRead PDF manual

Performs parallel web crawling and web scraping. It is designed to crawl, parse and store web pages to produce data that can be directly used for analysis application. For details see Khalil and Fakir (2017) <DOI:10.1016/j.softx.2017.04.004>.

  • Maintainer: Salim Khalil
  • License: GPL (>= 2)
  • Last published: 2018-11-11

Useful links