parse_robotstxt function

function parsing robots.txt

robotstxt packageRead PDF manual
  • Maintainer: Pedro Baltazar
  • License: MIT + file LICENSE
  • Last published: 2024-08-29

Useful links