Robots controls ยท Updated October 21, 2026
WTNMarket robots directive audit for public and admin pages
Audit robots.txt, meta robots and response headers so public WTNMarket guides stay crawlable while admin routes remain excluded. This guide treats WTN Market and WTNMarket search terms as research labels, not as proof that a destination is official or safe.
Map every control layer
A WTNMarket page can be affected by robots.txt, an HTML meta robots tag and an HTTP response header. Record all three rather than checking only one.
The controls serve different purposes: crawl guidance is not the same as an instruction to exclude a page from search results.
Protect administrative routes
Admin, setup and recovery screens should not appear in the XML sitemap and should return noindex protections. They should also require authentication rather than relying on robots rules for privacy.
Test both the route and its response headers after deployment.
Keep public guides discoverable
Canonical article pages should return 200, allow crawling and use index,follow when publication is intended. Their canonical URL must match the sitemap entry.
A public article blocked in robots.txt cannot be reliably evaluated by the crawler that received the block.
Automate conflict detection
Fail the release when a sitemap URL is noindex, blocked, redirected or noncanonical. Run the same checks against the deployed host because edge rules can change the response.
After corrections, submit only the affected canonical URLs and monitor Bing URL Inspection.
Frequently asked questions
Does robots.txt remove a page from search results?
Not necessarily. It controls crawling, while noindex must be read from a permitted response.
Should admin pages be listed in the sitemap?
No. Administrative and recovery routes should stay outside the public sitemap.