Function reference • robotstxt

All functions

as.list(<robotstxt_text>): Method as.list() for class robotstxt_text

fix_url(): fix_url

get_robotstxt(): downloading robots.txt file

rt_last_http get_robotstxt_http_get(): storage for http request response objects

get_robotstxts(): function to get multiple robotstxt files

guess_domain(): function guessing domain from path

http_domain_changed(): http_domain_changed

http_subdomain_changed(): http_subdomain_changed

http_was_redirected(): http_was_redirected

is_suspect_robotstxt(): is_suspect_robotstxt

is_valid_robotstxt(): function that checks if file is valid / parsable robots.txt file

list_merge(): Merge a number of named lists in sequential order

null_to_defeault(): null_to_defeault

parse_robotstxt(): function parsing robots.txt

paths_allowed(): check if a bot has permissions to access page(s)

paths_allowed_worker_spiderbar(): paths_allowed_worker spiderbar flavor

%>%: re-export magrittr pipe operator

print(<robotstxt>): printing robotstxt

print(<robotstxt_text>): printing robotstxt_text

remove_domain(): function to remove domain from path

request_handler_handler(): request_handler_handler

robotstxt(): Generate a representations of a robots.txt file

rt_cache: get_robotstxt() cache

rt_request_handler() on_server_error_default on_client_error_default on_not_found_default on_redirect_default on_domain_change_default on_sub_domain_change_default on_file_type_mismatch_default on_suspect_content_default: rt_request_handler