is_suspect_robotstxt

function that checks if file is valid / parsable robots.txt file

Usage

is_suspect_robotstxt(text)

Arguments

text

content of a robots.txt file provides as character vector