can_fetch function

Test URL paths against a robxp robots.txt object

spiderbar packageRead PDF manual
  • Maintainer: Bob Rudis
  • License: MIT + file LICENSE
  • Last published: 2023-02-11

Useful links