How To Build a Web Scraping MCP Server with Python

By · · AI Engineering

I have been working with LLMs since 2023, and around December 2024, I got really interested in the Model Context Protocol (MCP). If you haven't come across it yet, MCP is basically a way for AI models to call external tools. You define what a tool does, what inputs it takes, and the AI decides when to use it.

I kept running into the same problem: I would copy text from a web page, paste it into a prompt, and repeat that ten times in one session. At some point I thought, I should just write an MCP server that does this for me. Let the AI fetch pages itself.

So that is what this post is about. The server is small, maybe a hundred lines, and it does two things: grabs text from a page and pulls out links.

What we are building

A small MCP server with two tools:

That is the whole thing.

Prerequisites

You need Python 3.10+ and three packages:

pip install mcp httpx beautifulsoup4

Project structure

web-scraper-mcp/
├── server.py
└── requirements.txt

Everything fits in one file.

The full server code

GitHub Gist: https://gist.github.com/encryptedtouhid/07f960a7845a10d912fb087cdcb4cb01

Here is the complete server:

import asyncio
import httpx
from bs4 import BeautifulSoup
from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import Tool, TextContent


app = Server("web-scraper")


async def fetch_page(url: str) -> str:
    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; MCPScraper/1.0)"
    }
    async with httpx.AsyncClient(follow_redirects=True, timeout=30.0) as client:
        response = await client.get(url, headers=headers)
        response.raise_for_status()
        return response.text


def extract_text(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header"]):
        tag.decompose()
    text = soup.get_text(separator="\n", strip=True)
    lines = [line.strip() for line in text.splitlines() if line.strip()]
    return "\n".join(lines)


def extract_links(html: str, base_url: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    links = []
    for anchor in soup.find_all("a", href=True):
        href = anchor["href"]
        if href.startswith("/"):
            href = base_url.rstrip("/") + href
        if href.startswith("http"):
            links.append({
                "text": anchor.get_text(strip=True),
                "url": href
            })
    return links


@app.list_tools()
async def list_tools() -> list[Tool]:
    return [
        Tool(
            name="scrape_url",
            description="Fetch a web page and return its text content with HTML tags removed",
            inputSchema={
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The URL of the web page to scrape"
                    }
                },
                "required": ["url"]
            }
        ),
        Tool(
            name="scrape_links",
            description="Fetch a web page and return all hyperlinks found on it",
            inputSchema={
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The URL of the web page to extract links from"
                    }
                },
                "required": ["url"]
            }
        )
    ]


@app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
    url = arguments.get("url", "")
    if not url:
        return [TextContent(type="text", text="Error: No URL provided")]

    try:
        html = await fetch_page(url)
    except httpx.HTTPStatusError as e:
        return [TextContent(type="text", text=f"HTTP error: {e.response.status_code}")]
    except Exception as e:
        return [TextContent(type="text", text=f"Failed to fetch URL: {str(e)}")]

    if name == "scrape_url":
        text = extract_text(html)
        if not text:
            return [TextContent(type="text", text="No readable content found on the page")]
        return [TextContent(type="text", text=text)]

    elif name == "scrape_links":
        links = extract_links(html, url)
        if not links:
            return [TextContent(type="text", text="No links found on the page")]
        result = "\n".join(f"- [{l['text']}]({l['url']})" for l in links)
        return [TextContent(type="text", text=result)]

    return [TextContent(type="text", text=f"Unknown tool: {name}")]


async def main():
    async with stdio_server() as (read_stream, write_stream):
        await app.run(read_stream, write_stream, app.create_initialization_options())


if __name__ == "__main__":
    asyncio.run(main())

Let me walk through what each part does.

How it works

Fetching pages

fetch_page uses httpx to make async HTTP requests. I set a custom user-agent header because some sites will block you without one. It follows redirects and times out after 30 seconds, which is enough for most pages.

Extracting text

extract_text is where the scraping happens. BeautifulSoup parses the raw HTML, I throw away scripts, stylesheets, nav bars, headers, and footers, then pull out just the text. The output is plain text that an LLM can read without tripping over HTML tags.

Extracting links

extract_links finds every anchor tag with an href, turns relative URLs into absolute ones, and gives back a list with the link text and URL. Useful when you want the AI to follow links and explore a site.

The MCP wiring

list_tools tells the client what tools exist and what arguments they take. call_tool receives the requests and routes them to the right function. Communication happens over stdin/stdout through stdio_server, so there is no HTTP server to set up.

Using it with Claude

Claude Desktop and Claude Code both support MCP servers. Open your claude_desktop_config.json and add the server:

{
  "mcpServers": {
    "web-scraper": {
      "command": "python",
      "args": ["path/to/server.py"]
    }
  }
}

On macOS this file is at ~/Library/Application Support/Claude/claude_desktop_config.json. On Windows it is %APPDATA%\Claude\claude_desktop_config.json.

Save the file, restart Claude Desktop, and you should see the web-scraper tools in the tools menu. After that you can just say "scrape the content from https://example.com" and Claude will call your server.

If you use Claude Code (the CLI), add the server in your project's .claude/settings.json or configure it globally. It works the same way.

Using it with ChatGPT

ChatGPT supports MCP servers through its desktop app too. Go to Settings, then MCP Servers, and add a new one:

Once it is added, ChatGPT shows the available tools and you can ask it to scrape pages the same way you would with Claude. The server code is identical since MCP is a standard protocol. ChatGPT might handle tool results a little differently in the UI, but nothing you need to worry about on the server side.

Testing with MCP Inspector

If you want to try the server before connecting it to anything, the MCP SDK has an inspector tool built in:

mcp dev server.py

This opens a web UI where you can call the tools manually and see the raw responses. I usually do this first to make sure everything is working before I hook it up to Claude or ChatGPT.

Where to go from here

I kept this intentionally bare bones. Some things I have been thinking about adding:

Adding a new tool is just another schema and handler. The client picks it up next time it connects.

Wrapping up

The whole server is under a hundred lines. MCP handles the communication, Python handles the scraping. What makes it worth building is that once this exists, the AI can use it alongside your other tools. I have been pairing mine with a file writer and a database tool, and that combination covers most of what I need day to day.