Fetch

What is fetch-mcp?

fetch-mcp is a flexible HTTP fetching Model Context Protocol (MCP) server designed to retrieve web content in various formats. It acts as a server that can fetch HTML, JSON, Markdown, or plaintext from specified URLs.

How to use fetch-mcp?

To use fetch-mcp, you first need to clone the repository, install dependencies (npm install), and build the server (npm run build). You can then run the server directly using npm start. For integration with a desktop application, you can configure it by adding a JSON snippet to your app's server configuration, specifying the command and arguments to run the dist/index.js file.

Key Features of fetch-mcp

fetch-mcp offers the following key features:

• Flexible Content Fetching: Fetches web content using the modern fetch API.
• Multiple Formats: Supports fetching content in HTML, JSON, plain text, and Markdown formats.
• Custom Headers: Allows inclusion of custom headers for requests.
• HTML Parsing: Utilizes JSDOM for robust HTML parsing and text extraction.
• Markdown Conversion: Employs TurndownService for converting HTML content to Markdown format.
• Tools: Provides specific tools like fetch_html, fetch_json, fetch_txt, and fetch_markdown for targeted content retrieval.

Use Cases of fetch-mcp

fetch-mcp is ideal for scenarios requiring on-demand fetching and transformation of web content. Potential use cases include:

• Data Scraping: Extracting data from websites in various formats for analysis or storage.
• Content Aggregation: Gathering content from multiple sources for display or processing.
• API Integration: Fetching JSON data from APIs for application use.
• Content Transformation: Converting web pages into plain text or Markdown for easier readability or specific applications.
• Automated Reporting: Collecting web content for automated report generation.

FAQ about fetch-mcp

Q: What types of content can fetch-mcp retrieve? A: fetch-mcp can retrieve HTML, JSON, plain text, and Markdown content.

Q: Does fetch-mcp support custom headers for requests? A: Yes, you can include custom headers in your fetch requests.

Q: How does fetch-mcp handle HTML parsing? A: It uses JSDOM for efficient HTML parsing and text extraction.

Q: Can fetch-mcp convert HTML to Markdown? A: Yes, it uses TurndownService to convert HTML content to Markdown format.

Q: Does fetch-mcp store any persistent resources? A: No, fetch-mcp is designed to fetch and transform web content on demand and does not provide any persistent resources.

Fetch MCP Server

This MCP server provides functionality to fetch web content in various formats, including HTML, JSON, plain text, and Markdown.

Components

Tools

• fetch_html

  • Fetch a website and return the content as HTML
  • Input:
    • url (string, required): URL of the website to fetch
    • headers (object, optional): Custom headers to include in the request
  • Returns the raw HTML content of the webpage

• fetch_json

  • Fetch a JSON file from a URL
  • Input:
    • url (string, required): URL of the JSON to fetch
    • headers (object, optional): Custom headers to include in the request
  • Returns the parsed JSON content

• fetch_txt

  • Fetch a website and return the content as plain text (no HTML)
  • Input:
    • url (string, required): URL of the website to fetch
    • headers (object, optional): Custom headers to include in the request
  • Returns the text content of the webpage with HTML tags, scripts, and styles removed

• fetch_markdown

  • Fetch a website and return the content as Markdown
  • Input:
    • url (string, required): URL of the website to fetch
    • headers (object, optional): Custom headers to include in the request
  • Returns the content of the webpage converted to Markdown format

Resources

This server does not provide any persistent resources. It's designed to fetch and transform web content on demand.

Getting started

1. Clone the repository
2. Install dependencies: npm install
3. Build the server: npm run build

Usage

To use the server, you can run it directly:

npm start

This will start the Fetch MCP Server running on stdio.

Usage with Desktop App

To integrate this server with a desktop app, add the following to your app's server configuration:

{
  "mcpServers": {
    "fetch": {
      "command": "node",
      "args": [
        "{ABSOLUTE PATH TO FILE HERE}/dist/index.js"
      ]
    }
  }
}

Features

• Fetches web content using modern fetch API
• Supports custom headers for requests
• Provides content in multiple formats: HTML, JSON, plain text, and Markdown
• Uses JSDOM for HTML parsing and text extraction
• Uses TurndownService for HTML to Markdown conversion

Development

• Run npm run dev to start the TypeScript compiler in watch mode
• Use npm test to run the test suite

License

This project is licensed under the MIT License.