Rquest

What is mcp-rquest?

mcp-rquest is a Model Context Protocol (MCP) server designed to enhance the capabilities of large language models (LLMs) like Claude. It provides advanced HTTP request functionalities, allowing LLMs to interact with websites more realistically by emulating browser behavior with accurate TLS/JA3/JA4 fingerprints. This helps in bypassing anti-bot measures. Additionally, mcp-rquest supports converting PDF and HTML documents into Markdown format, making them easier for LLMs to process.

How to use mcp-rquest?

mcp-rquest can be installed using pip or uv. After installation, it can be run as a Python script. For integration with Claude.app, you can configure it in your Claude settings by specifying the command and arguments for mcp-rquest.

Installation:

• Using uv (recommended): No specific installation needed, use uvx to directly run mcp-rquest.
• Using pip: pip install mcp-rquest then python -m mcp_rquest to run.

Configuration for Claude.app:

Add the following to your Claude settings JSON, adjusting the command and args based on your installation method (uvx, pip, or pipx):

{
  "mcpServers": {
    "http-rquest": {
      "command": "uvx",
      "args": ["mcp-rquest"]
    }
  }
}

Key Features of mcp-rquest

• Complete HTTP Methods: Supports GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS, and TRACE.
• Browser Fingerprinting: Provides accurate TLS, JA3/JA4, and HTTP/2 browser fingerprints for realistic browser emulation.
• Content Handling: Automatically handles large responses, converts HTML and PDF to Markdown, and securely stores responses.
• Authentication Support: Basic, Bearer, and custom authentication methods.
• Request Customization: Allows customization of headers, cookies, redirects, form data, JSON payloads, and query parameters.
• SSL Security: Uses BoringSSL for secure connections with realistic browser fingerprints.
• PDF Support: Seamless PDF to Markdown conversion using the Marker library, with automatic detection and optimized performance.

Use Cases of mcp-rquest

• Bypassing Anti-Bot Measures: LLMs can interact with websites that employ anti-bot mechanisms by appearing as legitimate browser traffic.
• Enhanced Web Interaction for LLMs: Enables LLMs to perform more natural and effective web requests.
• Document Processing for LLMs: Converts complex PDF and HTML documents into a simplified Markdown format, making it easier for LLMs to extract and process information.
• Automated Data Collection: Facilitates automated data collection from websites that require realistic browser behavior.

FAQ from mcp-rquest

• What is the purpose of browser emulation? Browser emulation helps in bypassing bot detection and accessing content normally available only to standard browsers by providing realistic browser fingerprints.
• Which browser fingerprints are supported? Supported browser fingerprints include Chrome (multiple versions), Firefox, Safari (including iOS and iPad versions), Edge, and OkHttp.
• How does mcp-rquest handle PDF conversion? mcp-rquest automatically detects PDF files, converts them to Markdown using the Marker library, and optimizes performance by pre-downloading models during installation.
• Can I retrieve large responses? Yes, you can use the get_stored_response tool to retrieve stored large responses, optionally by line range.
• How do I convert HTML or PDF responses to Markdown? Use the get_stored_response_with_markdown tool, which works for both HTML and PDF files.

mcp-rquest

A Model Context Protocol (MCP) server that provides advanced HTTP request capabilities for Claude and other LLMs. Built on rquest, this server enables realistic browser emulation with accurate TLS/JA3/JA4 fingerprints, allowing models to interact with websites more naturally and bypass common anti-bot measures. It also supports converting PDF and HTML documents to Markdown for easier processing by LLMs.

Features

• Complete HTTP Methods: Support for GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS, and TRACE
• Browser Fingerprinting: Accurate TLS, JA3/JA4, and HTTP/2 browser fingerprints
• Content Handling:
  • Automatic handling of large responses with token counting
  • HTML to Markdown conversion for better LLM processing
  • PDF to Markdown conversion using the Marker library
  • Secure storage of responses in system temporary directories
• Authentication Support: Basic, Bearer, and custom authentication methods
• Request Customization:
  • Headers, cookies, redirects
  • Form data, JSON payloads, multipart/form-data
  • Query parameters
• SSL Security: Uses BoringSSL for secure connections with realistic browser fingerprints

Available Tools

• HTTP Request Tools:

  • http_get - Perform GET requests with optional parameters
  • http_post - Submit data via POST requests
  • http_put - Update resources with PUT requests
  • http_delete - Remove resources with DELETE requests
  • http_patch - Partially update resources
  • http_head - Retrieve only headers from a resource
  • http_options - Retrieve options for a resource
  • http_trace - Diagnostic request tracing

• Response Handling Tools:

  • get_stored_response - Retrieve stored large responses, optionally by line range
  • get_stored_response_with_markdown - Convert HTML or PDF responses to Markdown format for better LLM processing
  • get_model_state - Get the current state of the PDF models loading process
  • restart_model_loading - Restart the PDF models loading process if it failed or got stuck

PDF Support

mcp-rquest now supports PDF to Markdown conversion, allowing you to download PDF files and convert them to Markdown format that's easy for LLMs to process:

1. Automatic PDF Detection: PDF files are automatically detected based on content type
2. Seamless Conversion: The same get_stored_response_with_markdown tool works for both HTML and PDF files
3. High-Quality Conversion: Uses the Marker library for accurate PDF to Markdown transformation
4. Optimized Performance: Models are pre-downloaded during package installation to avoid delays during request processing

Installation

Using uv (recommended)

When using uv no specific installation is needed. We will use uvx to directly run mcp-rquest.

Using pip

Alternatively you can install mcp-rquest via pip:

pip install mcp-rquest

After installation, you can run it as a script using:

python -m mcp_rquest

Configuration

Configure for Claude.app

Add to your Claude settings:

Using uvx:

{
  "mcpServers": {
    "http-rquest": {
      "command": "uvx",
      "args": ["mcp-rquest"]
    }
  }
}

Using pip:

{
  "mcpServers": {
    "http-rquest": {
      "command": "python",
      "args": ["-m", "mcp_rquest"]
    }
  }
}

Using pipx:

{
  "mcpServers": {
    "http-rquest": {
      "command": "pipx",
      "args": ["run", "mcp-rquest"]
    }
  }
}

Browser Emulation

mcp-rquest leverages rquest's powerful browser emulation capabilities to provide realistic browser fingerprints, which helps bypass bot detection and access content normally available only to standard browsers. Supported browser fingerprints include:

• Chrome (multiple versions)
• Firefox
• Safari (including iOS and iPad versions)
• Edge
• OkHttp

This ensures that requests sent through mcp-rquest appear as legitimate browser traffic rather than bot requests.

Development

Setting up a Development Environment

1. Clone the repository
2. Create a virtual environment using uv:

   uv venv
   

3. Activate the virtual environment:

   # Unix/macOS
   source .venv/bin/activate
   # Windows
   .venv\Scripts\activate
   

4. Install development dependencies:

   uv pip install -e ".[dev]"
   

Acknowledgements

• This project is built on top of rquest, which provides the advanced HTTP client with browser fingerprinting capabilities.
• rquest is based on a fork of reqwest.