Validate if a file is valid / parsable robots.txt file — is_valid_robotstxt • robotstxt

Validate if a file is valid / parsable robots.txt file

Usage

is_valid_robotstxt(text, check_strickt_ascii = FALSE)

Arguments

text: content of a robots.txt file provided as character vector
check_strickt_ascii: whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII