Fetches and parses a site's robots.txt, returning rules, sitemaps, crawl delays, and block status.
Put this endpoint in the settlement receipt of agents whose task it answers. You pay only when the recommendation converts on-chain.