Docs Fetch MCP Server
A Model Context Protocol (MCP) server for fetching web content with recursive exploration capabilities. This server enables LLMs to autonomously explore web pages and documentation to learn about specific topics.
Overview
The Docs Fetch MCP Server provides a simple but powerful way for LLMs to retrieve and explore web content. It enables:
- Fetching clean, readable content from any web page
- Recursive exploration of linked pages up to a specified depth
- Same-domain link traversal to gather comprehensive information
- Smart filtering of navigation links to focus on content-rich pages
This tool is particularly useful when users want an LLM to learn about a specific topic by exploring documentation or web content.
Features
- **Content Extraction**: Cleanly extracts the main content from web pages, removing distractions like navigation, ads, and irrelevant elements
- **Link Analysis**: Identifies and extracts links from the page, assessing their relevance
- **Recursive Exploration**: Follows links to related content within the same domain, up to a specified depth
- **Parallel Processing**: Efficiently crawls content with concurrent requests and proper error handling
- **Robust Error Handling**: Gracefully handles network issues, timeouts, and malformed pages
- **Dual-Strategy Approach**: Uses fast axios requests first with puppeteer as a fallback for more complex pages
- **Timeout Prevention**: Implements global timeout handling to ensure reliable operation within MCP time limits
- **Partial Results**: Returns available content even when some pages fail to load completely
Usage
The server exposes a single MCP tool:
`fetch_doc_content`
Fetches web page content with the ability to explore linked pages up to a specified depth.
**Parameters:**
- `url` (string, required): URL of the web page to fetch
- `depth` (number, optional, default: 1): Maximum depth of directory/link exploration (1-5)
**Returns:**
Installation
- Clone this repository:
- Install dependencies:
- Build the project:
- Configure your MCP settings in your Claude Client:
Dependencies
- `@modelcontextprotocol/sdk`: MCP server SDK
- `puppeteer`: Headless browser for web page interaction
- `axios`: HTTP client for making requests
Development
To run the server in development mode:
License
MIT