Parse a robots.txt file

Parse a robots.txt file

Usage

parse_robotstxt(txt)

Arguments

txt

content of the robots.txt file

Value

a named list with useragents, comments, permissions, sitemap