mcp-server-fetch-python MCP Server: Advanced Web Content Fetching and Transformation

Overview: What is mcp-server-fetch-python MCP Server?

mcp-server-fetch-python is an MCP (Model Context Protocol) server designed to fetch, process, and transform web content into various formats. This powerful toolset supports modern web applications by extracting and rendering content directly from URLs with comprehensive browser emulation capabilities, ensuring that JavaScript-rendered elements are properly handled. By integrating mcp-server-fetch-python with AI applications like Claude Desktop, the system can provide rich data sources for natural language processing (NLP) models, enhancing their ability to understand complex web content.

🔧 Core Features & MCP Capabilities

mcp-server-fetch-python offers four specialized tools that cater to different use cases of web content extraction and transformation:

1. get-raw-text: This tool fetches raw text data from URLs without any browser rendering, making it ideal for structured data formats or scenarios where fast access is required.
2. get-rendered-html: Utilizes a headless browser to render the entire page fully, including JavaScript-generated content and media files. This is essential for modern web applications that rely on client-side JavaScript execution.
3. get-markdown: Converts web pages into well-formatted Markdown documents, preserving structural elements while ensuring readability.
4. get-markdown-from-media: Extracts text from media files such as images and videos using AI-powered techniques through computer vision and OCR (Optical Character Recognition).

Each tool is designed to handle specific tasks efficiently, providing a robust toolkit for developers working with web content and MCP clients.

⚙️ MCP Architecture & Protocol Implementation

The architecture of mcp-server-fetch-python aligns closely with the Model Context Protocol (MCP). It acts as an adapter that standardizes interactions between AI applications and diverse data sources. The server communicates using a standardized protocol, making it easier for MCP clients to integrate seamlessly.

Mermaid Diagram: MCP Protocol Flow

graph TD
    A[AI Application] -->|MCP Client| B[MCP Protocol]
    B --> C[MCP Server]
    C --> D[Data Source/Tool]
    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style D fill:#e8f5e8

Mermaid Diagram: Data Architecture

graph TD
    A[Client] -->|MCP Request| B[MCP Server]
    B --> C[Data Source/Tool]
    C --> D[MCP Response]
    D --> E[Client]
    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style D fill:#e8f5e8

🚀 Getting Started with Installation

To get started, you can install and run mcp-server-fetch-python locally or configure it as a remote server for your AI application.

Local Installation

1. Clone the Repository:

   git clone https://github.com/tatn/mcp-server-fetch-python.git
   cd mcp-server-fetch-python
   uv sync
   uv build
   

2. Configure MCP Clients: Add the following configuration to your Claude Desktop or similar MCP client’s settings file:

"mcpServers": {
  "mcp-server-fetch-python": {
    "command": "uv",
    "args": [
      "--directory",
      "path\\to\\mcp-server-fetch-python",  # Replace with actual path to the cloned repository
      "run",
      "mcp-server-fetch-python"
    ],
    "env": {
        "OPENAI_API_KEY": "sk-****",
        "PYTHONIOENCODING": "utf-8",
        "MODEL_NAME": "gpt-4o",        
    }
  }
}

💡 Key Use Cases in AI Workflows

Use Case 1: Content Aggregation for Chatbot Training

Suppose you are developing a chatbot that requires extensive data to learn about various online products. You can use mcp-server-fetch-python to scrape details from e-commerce website product pages and convert the extracted content into structured Markdown files.

import mcp_server_fetch_python

url_list = [
    "https://example.com/product1",
    "https://example.com/product2"
]

for url in url_list:
    text_content = mcp_server_fetch_python.get_markdown_from_url(url)
    with open(f"data/{url.split('/')[-1]}.md", 'w') as f:
        f.write(text_content)

Use Case 2: Automated Web App Testing

If your web application has dynamic elements that require JavaScript rendering, use get-rendered-html to simulate a user's interaction and check the output against expected results.

import mcp_server_fetch_python

rendered_html = mcp_server_fetch_python.get_rendered_html("https://example.com/testing-page")
print(rendered_html)

🔌 Integration with MCP Clients

mcp-server-fetch-python is fully compatible with several MCP clients, including Claude Desktop, Continue, and Cursor. The table below summarizes the compatibility status for each client.

MCP Client Compatibility Matrix

MCP Client	Resources	Tools	Prompts	Status
Claude Desktop	✅	✅	✅	Full Support
Continue	✅	✅	❌	Limited Tool Support
Cursor	❌	✅	❌	Tools Only

📊 Performance & Compatibility Matrix

mcp-server-fetch-python is designed to run efficiently and handle a wide range of HTTP requests, ensuring compatibility with modern web technologies. It supports multiple tools and can be easily extended or modified according to specific requirements.

🛠️ Advanced Configuration & Security

You can further customize mcp-server-fetch-python by setting environment variables for improved security and performance:

• OPENAI_API_KEY: Required if using get-markdown-from-media tool.
• PYTHONIOENCODING: Set to "utf-8" to handle character encoding issues.

Additionally, you can adjust the model name used by setting MODEL_NAME. The default value is "gpt-4o" but can be changed as needed.

❓ Frequently Asked Questions (FAQ)

1. Can I use mcp-server-fetch-python with Cursor?

   • Yes, it's compatible with tools in Cursor, though the client may not support all prompts.

2. Is it safe to use my API key in environment variables?

   • Yes, as long as there is minimal exposure and proper access controls are in place.

3. Does mcp-server-fetch-python support streaming output for large web pages?

   • Not explicitly, but you can implement pagination or chunking techniques to handle large content.

4. Can I integrate custom data sources with mcp-server-fetch-python?

   • Yes, by modifying the codebase and adding custom tools as necessary.

5. How do I troubleshoot errors related to JavaScript rendering in headless browsers?

   • Check network requests for missing resources or permission issues that could block execution of JavaScript.

👨‍💻 Development & Contribution Guidelines

Contributors can join efforts to improve mcp-server-fetch-python by:

• Fixing bugs and enhancing performance.
• Adding new tools and features based on user feedback.
• Improving documentation and usability.

New contributors are encouraged to review the existing issues and pull requests before beginning their work. Fork the repository, make changes, and submit a pull request for consideration.

🌐 MCP Ecosystem & Resources

Explore more about Model Context Protocol (MCP) by visiting the official website. Stay updated with the latest developments in machine learning frameworks and tools via the official GitHub repository.