Skip to contents

function that checks if file is valid / parsable robots.txt file

Source: R/is_valid_robotstxt.R
is_valid_robotstxt.Rd

function that checks if file is valid / parsable robots.txt file

Usage

is_valid_robotstxt(text, check_strickt_ascii = FALSE)

Arguments

text

content of a robots.txt file provides as character vector

check_strickt_ascii

whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII