AI Crawler Checker — Robots.txt Policy for Six Bots

Overview

AI Crawler Checker — Robots.txt Policy for Six Bots is an MCP server that checks declared robots.txt permissions for six AI crawlers at specified URL paths. Instead of manually testing crawler access policies, you can query this server as a callable tool through any MCP client, getting permission status directly in your workflow.

This is a hosted remote server accessible at https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check. It requires an API Key for authentication and runs at version v1.0.2. No local installation is needed — simply point your MCP client to the endpoint and begin checking crawler policies.

AI Crawler Checker sits in the Web Scraping category on mcp.site and works with MCP clients including Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, and Zed. If you maintain this server, claim this listing to verify ownership and keep details current.

Installation

Settings → Connectors → Add custom connector
Name: AI Crawler Checker — Robots.txt Policy for Six Bots
URL:  https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check