What MCP Is, Briefly
If you've been poking around AI tooling in the last year, you've probably run into the Model Context Protocol. Anthropic open-sourced it, and the idea is simple: instead of wiring every AI assistant to every data source with custom glue code that breaks the moment someone sneezes, you build a server that speaks one protocol and any MCP client can talk to it.
The architecture is client-server over JSON-RPC. Your AI assistant (Claude, whatever) is the client. Your code is the server. They pass messages back and forth over stdio or HTTP/SSE. The server advertises what it can do, and the client can call those tools at will.
What sold me was the tool discovery model. Your server says "I have a tool called search_web, here's its input schema," and the client can present that to the user and invoke it when needed. No hardcoded function calls. No "but it only works in this one chat interface."
Setting Up
Let me save you the existential dread of realizing you've somehow accumulated twelve Python projects all using different dependency managers. Here's what you actually need:
python -m venv mcp-env
source mcp-env/bin/activate
pip install "mcp<2" requests beautifulsoup4 playwright lxml async-lru
playwright install chromium
I used pip here because it's what most people have. If you're fancy and use uv, the incantation is similar and faster. The worst part was playwright install chromium taking forever because it downloads an actual browser. Not the protocol's fault. That's on me for deciding web scraping was a good idea.
The <2 pin is not optional. Everything below uses the low-level Server decorator API, and the 2.x SDK dropped it (FastMCP was renamed MCPServer in the same release). On a plain pip install mcp today the very first snippet dies with AttributeError: 'Server' object has no attribute 'list_tools' before it prints anything.
How I Accidentally Built a Minimal MCP Server
I started simple. The MCP Python SDK gives you a Server class, you register tool handlers with decorators, and you run it over stdio. I wanted to see if I could get a tool to appear in my chat client. Low ambition, high reward.
import asyncio
from mcp.server import Server
from mcp.types import Tool, TextContent
import mcp.server.stdio
app = Server("websearch-scraper")
@app.list_tools()
async def list_tools() -> list[Tool]:
return [
Tool(
name="search_web",
description="Search the web for information",
inputSchema={
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search query"
},
"max_results": {
"type": "number",
"description": "Maximum results to return",
"default": 5
}
},
"required": ["query"]
}
)
]
@app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if name == "search_web":
query = arguments["query"]
max_results = arguments.get("max_results", 5)
results = await perform_web_search(query, max_results)
return [TextContent(
type="text",
text=f"Search results for '{query}':\n\n{results}"
)]
raise ValueError(f"Unknown tool: {name}")
async def perform_web_search(query: str, max_results: int) -> str:
return f"Found {max_results} results for: {query}"
async def main():
async with mcp.server.stdio.stdio_server() as (read_stream, write_stream):
await app.run(
read_stream,
write_stream,
app.create_initialization_options()
)
if __name__ == "__main__":
asyncio.run(main())
The server starts, the client discovers the tool, and if you call search_web, it tells you it found results without actually finding any. Like a consultant.
app.list_tools() tells clients what you offer, app.call_tool() routes requests to your logic, and stdio_server() handles the stdin/stdout dance so your client can talk to your server over a subprocess pipe. Simple enough that I felt dangerous.
The DuckDuckGo Search That Made Me Feel Like a Hacker
DuckDuckGo has an HTML endpoint that works for free. No API key, no rate limit horror stories at small scale. Just HTML scraping, the old fashioned way.
import asyncio
import requests
from bs4 import BeautifulSoup
from urllib.parse import quote_plus
async def search_duckduckgo(query: str, max_results: int = 5) -> list[dict]:
url = f"https://html.duckduckgo.com/html/?q={quote_plus(query)}"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
try:
response = await asyncio.to_thread(
requests.get, url, headers=headers, timeout=10
)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
results = []
for result in soup.select('.result')[:max_results]:
title = result.select_one('.result__title')
snippet = result.select_one('.result__snippet')
if title and snippet:
results.append({
"title": title.get_text(strip=True),
"snippet": snippet.get_text(strip=True),
})
return results
except Exception as e:
raise Exception(f"Search failed: {str(e)}")
That asyncio.to_thread is not decoration. requests is synchronous, so calling it directly inside an async def parks the whole event loop for the length of the HTTP round trip, and every other tool call your server is serving stops dead. Pushing it onto a worker thread keeps the coroutine genuinely concurrent. (httpx.AsyncClient is the tidier answer if you are willing to swap the dependency.)
The quote_plus on the query handles spaces and special characters. The User-Agent header keeps DuckDuckGo from thinking you're a bot (you are a bot, but it's polite to pretend otherwise). The CSS selectors target the specific elements DuckDuckGo uses in its HTML search page.
This worked great for about two hours. Then I realized I also wanted the server to actually read the pages it found, not just return headlines.
Playwright: The Browser You Didn't Ask For
Web scraping with requests works fine for server-rendered pages. But most of the web is JavaScript-rendered these days, which means requests gets you an empty shell and a prayer. Playwright launches an actual headless Chromium browser and runs the page's JavaScript before handing you the DOM.
from playwright.async_api import async_playwright
async def scrape_webpage(url: str, selector: str = None) -> dict:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
try:
await page.goto(url, timeout=30000)
await page.wait_for_load_state('networkidle')
if selector:
element = await page.query_selector(selector)
content = await element.inner_text() if element else "Selector not found"
else:
content = await page.inner_text('body')
title = await page.title()
return {
"title": title,
"content": content[:5000],
"url": url,
"success": True
}
except Exception as e:
return {
"error": str(e),
"url": url,
"success": False
}
finally:
await browser.close()
Playwright downloads a full Chromium binary. Your Docker images get chunky. Your pip install takes longer. But it also means your MCP server can read JavaScript-rendered pages, which is most of the web.
The networkidle wait is the magic incantation. It tells Playwright to wait until there have been no network connections for at least 500 ms. Without it, you get the page before React has hydrated and your "content" is a loading spinner.
Configuration
Once your server exists, you need to tell your MCP client about it. For Claude Desktop, that means editing claude_desktop_config.json:
{
"mcpServers": {
"websearch-scraper": {
"command": "python",
"args": ["/path/to/your/mcp_server.py"],
"env": {}
}
}
}
The file lives at:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json - Linux:
~/.config/Claude/claude_desktop_config.json
I spent twenty minutes wondering why my server wasn't showing up before I realized I'd typed the path wrong.
The Mistakes I Made So You Don't Have To
Rate Limiting (Or: DuckDuckGo Will Block You)
My first "production" test involved running 50 searches in quick succession. DuckDuckGo responded with a polite 503 and then stopped responding entirely for five minutes. I deserved it.
import time
from collections import defaultdict
class RateLimiter:
def __init__(self, max_requests: int = 10, time_window: int = 60):
self.max_requests = max_requests
self.time_window = time_window
self.requests = defaultdict(list)
async def acquire(self, key: str):
now = time.time()
self.requests[key] = [
t for t in self.requests[key]
if now - t < self.time_window
]
if len(self.requests[key]) >= self.max_requests:
raise Exception("Rate limit exceeded")
self.requests[key].append(now)
The pattern cleans out old timestamps before checking the count. It's not fancy, but it keeps you from getting banned while you iterate.
Caching (Or: Why Are You Searching The Same Thing?)
The first time you search for "MCP Python tutorial" and scrape three pages, you don't notice. The tenth time, you start questioning your life choices.
from async_lru import alru_cache
@alru_cache(maxsize=100)
async def cached_search(query: str, max_results: int):
return await search_web(query, max_results)
functools.lru_cache is not async-aware in any Python version. Wrap a coroutine with it and you cache the coroutine object rather than the result, so the first call works and every later cache hit raises RuntimeError: cannot reuse already awaited coroutine. alru_cache from the async-lru package is the drop-in that actually works. It keeps the last 100 unique queries in memory. If you're running this as a long-lived server, you might want Redis or SQLite instead. But for a personal assistant, this is plenty.
Input Validation (Or: Someone Will URL Injection You)
Validate everything. I don't care how trusted your client is.
from urllib.parse import urlparse
def validate_url(url: str) -> bool:
try:
result = urlparse(url)
return all([result.scheme, result.netloc])
except:
return False
This catches javascript: URIs, bare paths, and the various creative inputs people try. The urlparse dance is Python's standard approach.
Making It Work Over HTTP
Stdio is great for local development. But if you want to deploy this somewhere that isn't your laptop, you need an HTTP transport. Streamable HTTP is the one to reach for: the old HTTP+SSE transport was deprecated in the 2025-03-26 spec revision and only survives for backward compatibility. If you do still need SSE, this is the wiring that works:
import mcp.server.sse
from starlette.responses import Response
async def main_sse():
from starlette.applications import Starlette
from starlette.routing import Mount, Route
sse = mcp.server.sse.SseServerTransport("/messages/")
async def handle_sse(request):
async with sse.connect_sse(
request.scope, request.receive, request._send
) as (read_stream, write_stream):
await app.run(
read_stream,
write_stream,
app.create_initialization_options()
)
return Response()
starlette_app = Starlette(
routes=[
Route("/sse", endpoint=handle_sse),
Mount("/messages/", app=sse.handle_post_message),
]
)
import uvicorn
await uvicorn.Server(
config=uvicorn.Config(starlette_app, host="0.0.0.0", port=8000)
).serve()
There is no get_server() on SseServerTransport. The two moving parts are connect_sse, which upgrades the GET into an event stream, and handle_post_message, which gets mounted separately so clients have somewhere to POST their messages. For anything new, use FastMCP(...).streamable_http_app() and skip SSE entirely (in the 2.x SDK that class is called MCPServer).
The Full Server
Here's the complete-ish server with search, scrape, validation, and error handling all wired up. You need auth, monitoring, and proper logging for production, but this is enough to be useful.
#!/usr/bin/env python3
"""MCP Server for Web Search and Scraping"""
import asyncio
import logging
from typing import Any
import requests
from bs4 import BeautifulSoup
from urllib.parse import quote_plus
from playwright.async_api import async_playwright
from mcp.server import Server
from mcp.types import Tool, TextContent
import mcp.server.stdio
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("websearch-scraper")
app = Server("websearch-scraper")
async def search_web(query: str, max_results: int = 5) -> str:
url = f"https://html.duckduckgo.com/html/?q={quote_plus(query)}"
headers = {"User-Agent": "Mozilla/5.0"}
try:
response = await asyncio.to_thread(
requests.get, url, headers=headers, timeout=10
)
soup = BeautifulSoup(response.text, 'html.parser')
results = []
for result in soup.select('.result')[:max_results]:
title = result.select_one('.result__title')
snippet = result.select_one('.result__snippet')
link = result.select_one('.result__url')
if title and snippet:
results.append({
"title": title.get_text(strip=True),
"snippet": snippet.get_text(strip=True),
"url": link.get_text(strip=True) if link else ""
})
if not results:
return "No results found."
formatted = [f"Found {len(results)} results for '{query}':\n"]
for i, r in enumerate(results, 1):
formatted.append(f"\n{i}. **{r['title']}**")
formatted.append(f" {r['snippet']}")
formatted.append(f" {r['url']}")
return "\n".join(formatted)
except Exception as e:
logger.error(f"Search failed: {e}")
return f"Search error: {str(e)}"
async def scrape_page(url: str, selector: str = None) -> str:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
try:
await page.goto(url, timeout=30000)
await page.wait_for_load_state('networkidle')
if selector:
el = await page.query_selector(selector)
content = await el.inner_text() if el else "Selector not found"
else:
content = await page.inner_text('body')
title = await page.title()
return f"# {title}\n\n{content[:8000]}"
except Exception as e:
return f"Scrape error for {url}: {str(e)}"
finally:
await browser.close()
@app.list_tools()
async def list_tools() -> list[Tool]:
return [
Tool(
name="search_web",
description="Search the web using DuckDuckGo",
inputSchema={
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"max_results": {"type": "number", "default": 5}
},
"required": ["query"]
}
),
Tool(
name="scrape_webpage",
description="Scrape content from a webpage",
inputSchema={
"type": "object",
"properties": {
"url": {"type": "string", "description": "URL to scrape"},
"selector": {
"type": "string",
"description": "Optional CSS selector for specific content"
}
},
"required": ["url"]
}
)
]
@app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
if name == "search_web":
results = await search_web(
arguments["query"],
arguments.get("max_results", 5)
)
return [TextContent(type="text", text=results)]
elif name == "scrape_webpage":
content = await scrape_page(
arguments["url"],
arguments.get("selector")
)
return [TextContent(type="text", text=content)]
raise ValueError(f"Unknown tool: {name}")
async def main():
async with mcp.server.stdio.stdio_server() as (read_stream, write_stream):
await app.run(
read_stream,
write_stream,
app.create_initialization_options()
)
if __name__ == "__main__":
asyncio.run(main())
Docker
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y wget && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN playwright install chromium
RUN playwright install-deps
COPY mcp_server.py .
CMD ["python", "mcp_server.py"]
This image is not small. Playwright + Chromium adds about 400MB. If that bothers you, stick with requests + BeautifulSoup only, or deploy the scraper as a separate service.
What I Wish I Knew Before Starting
-
The SDK is still young. The MCP Python SDK changes fast, and it already broke once for real: 2.0 removed the low-level
Serverdecorators used throughout this post and renamedFastMCPtoMCPServer. Pin your version (mcp<2for the code here) or expect things to shift under you. -
Stdio transport is surprisingly solid. I was skeptical about piping JSON-RPC over stdin/stdout, but it works. The SDK handles framing. Start with stdio and only add an HTTP transport when you need it.
-
LLMs lie about tool results. Your server will return results faithfully, but the LLM will occasionally make up URLs or claim a search found things it didn't. Not your server's fault. Just LLMs being LLMs.
-
You do not need a vector database. Every MCP tutorial I read insisted I needed embeddings and a vector store. For web search and scraping, you really don't. The LLM gets the text inline via the tool result. Keep it simple.
Wrapping Up
Building an MCP server in Python clicked for me in a way I didn't expect. The protocol handles most of the awkward plumbing. You write the logic. The Python SDK is solid, and the examples on GitHub will carry you the rest of the way.
Web search and scraping make a good first project because you get something useful fast. You build it, wire it into your chat client, and suddenly your AI assistant can fact-check itself against live search results. Watching it return real data instead of "I cannot browse the internet" makes the setup worth it.
MCP is still evolving. But a standard beats everyone rolling their own. And with Python, you can have something working in an afternoon.
I did. And I only cried twice.