MCP Webscan Server

Features

• **Page Fetching**: Convert web pages to Markdown for easy analysis

• **Link Extraction**: Extract and analyze links from web pages

• **Site Crawling**: Recursively crawl websites to discover content

• **Link Checking**: Identify broken links on web pages

• **Pattern Matching**: Find URLs matching specific patterns

• **Sitemap Generation**: Generate XML sitemaps for websites

Installation

Installing via Smithery

Manual Installation

Usage

Starting the Server

Available Tools

1. `fetch-page` - Fetches a web page and converts it to Markdown. - Parameters: - `url` (required): URL of the page to fetch. - `selector` (optional): CSS selector to target specific content.

2. `extract-links` - Extracts all links from a web page with their text. - Parameters: - `url` (required): URL of the page to analyze. - `baseUrl` (optional): Base URL to filter links. - `limit` (optional, default: 100): Maximum number of links to return.

3. `crawl-site` - Recursively crawls a website up to a specified depth. - Parameters: - `url` (required): Starting URL to crawl. - `maxDepth` (optional, default: 2): Maximum crawl depth (0-5).

4. `check-links` - Checks for broken links on a page. - Parameters: - `url` (required): URL to check links for.

5. `find-patterns` - Finds URLs matching a specific pattern. - Parameters: - `url` (required): URL to search in. - `pattern` (required): JavaScript-compatible regex pattern to match URLs against.

6. `generate-site-map` - Generates a simple XML sitemap by crawling. - Parameters: - `url` (required): Root URL for sitemap crawl. - `maxDepth` (optional, default: 2): Maximum crawl depth for discovering URLs (0-5). - `limit` (optional, default: 1000): Maximum number of URLs to include in the sitemap.

Example Usage with Claude Desktop

1. Configure the server in your Claude Desktop settings:

1. Use the tools in your conversations:

Development

Prerequisites

• Node.js >= 18

• npm

Project Structure (Post-Refactor)

Building

Development Mode

Error Handling

• Invalid parameters

• Network errors

• Content parsing errors

• URL validation

Contributing

1. Fork the repository

2. Create your feature branch (`git checkout -b feature/amazing-feature`)

3. Commit your changes (`git commit -m 'Add some amazing feature'`)

4. Push to the branch (`git push origin feature/amazing-feature`)

5. Open a Pull Request

License