R/rt_get_useragent.R
rt_get_useragent.Rd
Extract HTTP useragents from robots.txt
rt_get_useragent(txt)
content of the robots.txt file