setup

setup_robot_rules_parser

setup_robot_rules_storage

handle_response

send_request

allowed($request)

Returns 1 if request is allowed to be fetched, 0 if not. -1 and -2 are returned when there is a pending request to fetch /robots.txt

dispatch_pending_robots_txt

Dispatches requests that were pending because of a missing robots.txt entry

push_pending_robots_txt

Pushes a request in the wait queue for a robots.txt

parse_robot_rules

Parses the robot rule and stores it