mcp.site mcp.site Search… ⌘K Submit Server
servers/ web-scraping/ robotsgate
RO

RobotsGate

Unverified Claim & verify → Web Scraping

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Visit website ↗ Try Remote Server
Overview Installation

Overview

RobotsGate is an MCP server that lets you generate, validate, and check robots.txt rules for AI crawlers. Any MCP-compatible AI assistant can call RobotsGate's tools directly, letting the model decide when to use them as part of a conversation or agent workflow. The service is free to use, with an optional Pro key for advanced features.

RobotsGate runs as a remote hosted server. Connect your MCP client to https://robotsgate.mike-tusa.workers.dev/mcp and you're ready to go—no local installation needed. Authentication uses an API key. The current release is v0.3.1.

RobotsGate is listed under Web Scraping on mcp.site and works with Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, Zed, and other MCP clients. If you maintain RobotsGate, claim this listing to verify ownership, earn the Verified badge, and keep the details current.

▸Hosted endpoint — Streamable HTTP · https://robotsgate.mike-tusa.workers.dev/mcp

Installation

Settings → Connectors → Add custom connector
Name: RobotsGate
URL:  https://robotsgate.mike-tusa.workers.dev/mcp

Related servers

Gatetest icon
Gatetest
Web Scraping

120-module QA gate. Give Claude eyes (screenshots), ears (Sentry errors), and hands (verify fixes).

Added 2 months ago →
Nimrod Research icon
Nimrod Research
Web Scraping Remote

Web research for agents: quality-scored Google search, webpage extraction, and deep research.

★ 256 Added 2 months ago →
SH
Shiplight
Web Scraping

An MCP server that provides browser automation.

Added 2 months ago →
Scrapegraphai Scrapegraph Mcp icon
Scrapegraphai Scrapegraph Mcp
Web Scraping Remote

Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili….

★ 100 Added 2 months ago →
Pinkpixel Dev Web Scout Mcp icon
Pinkpixel Dev Web Scout Mcp
Web Scraping Remote

Search the web and extract clean, readable text from webpages. Process multiple URLs at once to sp….

★ 132 Added 2 months ago →
VE
Verant
Web Scraping Remote

AI proofreading for public web pages and full sites, with AI-verified corrections. Paid plans only.

Added 2 months ago →
>_ esc
↑↓ navigate↵ openesc close