storage for http request response objects
Source:R/get_robotstxt_http_get.R
get_robotstxt_http_get.Rd
storage for http request response objects
get_robotstxt() worker function to execute HTTP request
Usage
rt_last_http
get_robotstxt_http_get(
domain,
user_agent = utils::sessionInfo()$R.version$version.string,
ssl_verifypeer = 1
)
Arguments
- domain
the domain to get tobots.txt. file for
- user_agent
the user agent to use for HTTP request header
- ssl_verifypeer
analog to CURL option https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html -- and might help with robots.txt file retrieval in some cases