Puppeteer vision

What is Puppeteer Vision MCP Server?

Puppeteer Vision MCP Server is a tool designed for advanced web scraping. It leverages Puppeteer for browsing, Mozilla Readability for content extraction, and Turndown for Markdown conversion. Its key differentiator is the integration of AI vision capabilities to automatically handle interactive web elements like cookie consent banners, CAPTCHAs, paywalls, and other prompts that typically block content access.

How to Use Puppeteer Vision MCP Server?

The recommended way to use Puppeteer Vision MCP Server is via npx.

Prerequisites: Ensure Node.js and npm are installed.
Environment Setup: Provide your OPENAI_API_KEY (required) and other optional configurations (e.g., VISION_MODEL, API_BASE_URL, TRANSPORT_TYPE, DISABLE_HEADLESS) via a .env file or shell environment variables.
Run the Server: Execute npx -y puppeteer-vision-mcp-server in your terminal.

Alternatively, it can be integrated as an MCP tool within an LLM orchestrator by configuring it with the npx command and environment variables.

For development or contribution, you can clone the repository, install dependencies (npm install), build the project (npm run build), set up your .env file, and then run it with npm start or npm run dev.

Key Features of Puppeteer Vision MCP Server

AI-Powered Interaction: Automatically handles cookie consent, CAPTCHAs, newsletter prompts, paywalls, age verification, and interstitial ads using AI vision.
Web Scraping: Scrapes webpages using Puppeteer with stealth mode.
Content Extraction: Extracts main content with Mozilla's Readability.
Markdown Conversion: Converts HTML to well-formatted Markdown with special handling for code blocks and tables.
MCP Accessibility: Accessible via the Model Context Protocol.
Real-time Interaction View: Option to disable headless mode to view browser interaction.
Easy Consumption: Easily consumable as an npx package.
Flexible Communication Modes: Supports stdio (default), sse, and http communication modes.

Use Cases of Puppeteer Vision MCP Server

Automated Data Collection: Efficiently scrape data from websites that have dynamic content or interactive elements.
Content Archiving: Convert web pages into clean Markdown for documentation or archiving purposes, bypassing common obstructions.
Research and Analysis: Gather information from various online sources for research without manual intervention for pop-ups or login walls.
LLM Integration: Integrate with LLM orchestrators as a web scraping tool to provide real-time web content to language models.

FAQ about Puppeteer Vision MCP Server

Q: What is OPENAI_API_KEY used for? A: It is your API key for accessing the vision model, which is crucial for the AI-driven interaction capabilities.

Q: Can I see the browser interacting with the webpage? A: Yes, you can set DISABLE_HEADLESS=true in your environment variables to run the browser in visible mode.

Q: What are the different communication modes? A: The server supports stdio (default, direct process communication), sse (Server-Sent Events over HTTP), and http (Streamable HTTP transport with session management).

Q: How do I specify the vision model to use? A: You can set the VISION_MODEL environment variable. The default is gpt-4.1, but you can use any model with vision capabilities.

Q: What are the parameters for the scrape-webpage tool? A: The tool accepts url (required), autoInteract (optional, default: true), maxInteractionAttempts (optional, default: 3), and waitForNetworkIdle (optional, default: true).

Puppeteer vision MCP Server

This Model Context Protocol (MCP) server provides a tool for scraping webpages and converting them to markdown format using Puppeteer, Readability, and Turndown. It features AI-driven interaction capabilities to handle cookies, captchas, and other interactive elements automatically.

Now easily runnable via npx!

Features

Scrapes webpages using Puppeteer with stealth mode
Uses AI-powered interaction to automatically handle:
- Cookie consent banners
- CAPTCHAs
- Newsletter or subscription prompts
- Paywalls and login walls
- Age verification prompts
- Interstitial ads
- Any other interactive elements blocking content
Extracts main content with Mozilla's Readability
Converts HTML to well-formatted Markdown
Special handling for code blocks, tables, and other structured content
Accessible via the Model Context Protocol
Option to view browser interaction in real-time by disabling headless mode
Easily consumable as an npx package.

Quick Start with NPX

The recommended way to use this server is via npx, which ensures you're running the latest version without needing to clone or manually install.

Prerequisites: Ensure you have Node.js and npm installed.

Environment Setup: The server requires an OPENAI_API_KEY. You can provide this and other optional configurations in two ways:

.env file: Create a .env file in the directory where you will run the npx command.
Shell Environment Variables: Export the variables in your terminal session.

Example .env file or shell exports:

# Required
OPENAI_API_KEY=your_api_key_here

# Optional (defaults shown)
# VISION_MODEL=gpt-4.1
# API_BASE_URL=https://api.openai.com/v1   # Uncomment to override
# TRANSPORT_TYPE=stdio                     # Options: stdio, sse, http
# USE_SSE=true                             # Deprecated: use TRANSPORT_TYPE=sse instead
# PORT=3001                                # Only used in sse/http modes
# DISABLE_HEADLESS=true                    # Uncomment to see the browser in action

Run the Server: Open your terminal and run:
```
npx -y puppeteer-vision-mcp-server
```
- The -y flag automatically confirms any prompts from npx.
- This command will download (if not already cached) and execute the server.
- By default, it starts in stdio mode. Set TRANSPORT_TYPE=sse or TRANSPORT_TYPE=http for HTTP server modes.

Using as an MCP Tool with NPX

This server is designed to be integrated as a tool within an MCP-compatible LLM orchestrator. Here's an example configuration snippet:

{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "puppeteer-vision-mcp-server"],
      "env": {
        "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE",
        // Optional:
        // "VISION_MODEL": "gpt-4.1",
        // "API_BASE_URL": "https://api.example.com/v1",
        // "TRANSPORT_TYPE": "stdio", // or "sse" or "http"
        // "DISABLE_HEADLESS": "true" // To see the browser during operations
      }
    }
    // ... other MCP servers
  }
}

When configured this way, the MCP orchestrator will manage the lifecycle of the puppeteer-vision-mcp-server process.

Environment Configuration Details

Regardless of how you run the server (NPX or local development), it uses the following environment variables:

OPENAI_API_KEY: (Required) Your API key for accessing the vision model.
VISION_MODEL: (Optional) The model to use for vision analysis.
- Default: gpt-4.1
- Can be any model with vision capabilities.
API_BASE_URL: (Optional) Custom API endpoint URL.
- Use this to connect to alternative OpenAI-compatible providers (e.g., Together.ai, Groq, Anthropic, local deployments).
TRANSPORT_TYPE: (Optional) The transport protocol to use.
- Options: stdio (default), sse, http
- stdio: Direct process communication (recommended for most use cases)
- sse: Server-Sent Events over HTTP (legacy mode)
- http: Streamable HTTP transport with session management
USE_SSE: (Optional, deprecated) Set to true to enable SSE mode over HTTP.
- Deprecated: Use TRANSPORT_TYPE=sse instead.
PORT: (Optional) The port for the HTTP server in SSE or HTTP mode.
- Default: 3001.
DISABLE_HEADLESS: (Optional) Set to true to run the browser in visible mode.
- Default: false (browser runs in headless mode).

Communication Modes

The server supports three communication modes:

stdio (Default): Communicates via standard input/output.
- Perfect for direct integration with LLM tools that manage processes.
- Ideal for command-line usage and scripting.
- No HTTP server is started. This is the default mode.
SSE mode: Communicates via Server-Sent Events over HTTP.
- Enable by setting TRANSPORT_TYPE=sse in your environment.
- Starts an HTTP server on the specified PORT (default: 3001).
- Use when you need to connect to the tool over a network.
- Connect to: http://localhost:3001/sse
HTTP mode: Communicates via Streamable HTTP transport with session management.
- Enable by setting TRANSPORT_TYPE=http in your environment.
- Starts an HTTP server on the specified PORT (default: 3001).
- Supports full session management and resumable connections.
- Connect to: http://localhost:3001/mcp

Tool Usage (MCP Invocation)

The server provides a scrape-webpage tool.

Tool Parameters:

url (string, required): The URL of the webpage to scrape.
autoInteract (boolean, optional, default: true): Whether to automatically handle interactive elements.
maxInteractionAttempts (number, optional, default: 3): Maximum number of AI interaction attempts.
waitForNetworkIdle (boolean, optional, default: true): Whether to wait for network to be idle before processing.

Response Format:

The tool returns its result in a structured format:

content: An array containing a single text object with the raw markdown of the scraped webpage.
metadata: Contains additional information:
- message: Status message.
- success: Boolean indicating success.
- contentSize: Size of the content in characters (on success).

Example Success Response:

{
  "content": [
    {
      "type": "text",
      "text": "# Page Title\n\nThis is the content..."
    }
  ],
  "metadata": {
    "message": "Scraping successful",
    "success": true,
    "contentSize": 8734
  }
}

Example Error Response:

{
  "content": [
    {
      "type": "text",
      "text": ""
    }
  ],
  "metadata": {
    "message": "Error scraping webpage: Failed to load the URL",
    "success": false
  }
}

How It Works

AI-Driven Interaction

The system uses vision-capable AI models (configurable via VISION_MODEL and API_BASE_URL) to analyze screenshots of web pages and decide on actions like clicking, typing, or scrolling to bypass overlays and consent forms. This process repeats up to maxInteractionAttempts.

Content Extraction

After interactions, Mozilla's Readability extracts the main content, which is then sanitized and converted to Markdown using Turndown with custom rules for code blocks and tables.

Installation & Development (for Modifying the Code)

If you wish to contribute, modify the server, or run a local development version:

Clone the Repository:

git clone https://github.com/djannot/puppeteer-vision-mcp.git
cd puppeteer-vision-mcp

Install Dependencies:
```
npm install
```
Build the Project:
```
npm run build
```
Set Up Environment: Create a .env file in the project's root directory with your OPENAI_API_KEY and any other desired configurations (see "Environment Configuration Details" above).

Run for Development:

npm start # Starts the server using the local build

Or, for automatic rebuilding on changes:

npm run dev

Customization (for Developers)

You can modify the behavior of the scraper by editing:

src/ai/vision-analyzer.ts (analyzePageWithAI function): Customize the AI prompt.
src/ai/page-interactions.ts (executeAction function): Add new action types.
src/scrapers/webpage-scraper.ts (visitWebPage function): Change Puppeteer options.
src/utils/markdown-formatters.ts: Adjust Turndown rules for Markdown conversion.

Dependencies

Key dependencies include:

@modelcontextprotocol/sdk
puppeteer, puppeteer-extra
@mozilla/readability, jsdom
turndown, sanitize-html
openai (or compatible API for vision models)
express (for SSE mode)
zod

Puppeteer vision Overview