mcp.site mcp.site Search… ⌘K Submit Server
servers/ web-scraping/ robotstxt-ai-crawler-checker
RA

Robots.txt & AI Crawler Checker

Unverified Claim & verify → Web Scraping

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Visit website ↗ Try Remote Server
Overview Installation

Overview

Robots.txt & AI Crawler Checker is an MCP server that validates robots.txt files against RFC 9309, checks whether AI crawlers like GPTBot and ClaudeBot can access a site, and verifies sitemaps. Any MCP-compatible AI assistant can call its tools directly during conversation or agent runs.

This is a hosted remote server with no installation required — simply point your MCP client to https://robots-check.quietworklabs.workers.dev/mcp to connect. The current release is v1.0.0, and no authentication is needed.

Robots.txt & AI Crawler Checker appears under Web Scraping on mcp.site and works with MCP clients including Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, Zed and others. If you maintain this server, claim this listing to verify ownership, earn the Verified badge, and keep information up to date.

▸Hosted endpoint — Streamable HTTP · https://robots-check.quietworklabs.workers.dev/mcp

Installation

Settings → Connectors → Add custom connector
Name: Robots.txt & AI Crawler Checker
URL:  https://robots-check.quietworklabs.workers.dev/mcp

Related servers

Gatetest icon
Gatetest
Web Scraping

120-module QA gate. Give Claude eyes (screenshots), ears (Sentry errors), and hands (verify fixes).

Added 2 months ago →
Nimrod Research icon
Nimrod Research
Web Scraping Remote

Web research for agents: quality-scored Google search, webpage extraction, and deep research.

★ 256 Added 2 months ago →
SH
Shiplight
Web Scraping

An MCP server that provides browser automation.

Added 2 months ago →
Scrapegraphai Scrapegraph Mcp icon
Scrapegraphai Scrapegraph Mcp
Web Scraping Remote

Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili….

★ 100 Added 2 months ago →
Pinkpixel Dev Web Scout Mcp icon
Pinkpixel Dev Web Scout Mcp
Web Scraping Remote

Search the web and extract clean, readable text from webpages. Process multiple URLs at once to sp….

★ 132 Added 2 months ago →
VE
Verant
Web Scraping Remote

AI proofreading for public web pages and full sites, with AI-verified corrections. Paid plans only.

Added 2 months ago →
>_ esc
↑↓ navigate↵ openesc close