Website | Github | Docs

mcp-server-webcrawl

Bridge the gap between your web crawl and AI language models using Model Context Protocol (MCP). With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously. The server includes a full-text search interface with boolean support, resource filtering by type, HTTP status, and more.

mcp-server-webcrawl provides the LLM a complete menu with which to search your web content, and works with a variety of web crawlers:

• WARC

• wget

• InterroBot

• Katana

• SiteOne

mcp-server-webcrawl is free and open source, and requires Claude Desktop, Python (>=3.10). It is installed on the command line, via pip install:

pip install mcp_server_webcrawl

Contents:

• Installation
• Usage
• mcp_server_webcrawl

Indices and tables

• Index

• Module Index

• Search Page