Self-host Scrapling's MCP server for your agents
A private scraping box for you and your agents, with Scrapling's MCP server.
Shelley handles the setup. About 10 minutes.
Prefer a terminal?
curl -fsSL https://exe-staging.dev/deploy/scrapling/prompt.txt | ssh exe.dev new --prompt=/dev/stdinScrapling is a Python scraping library with more than 100,000 stars on GitHub: an HTTP client that impersonates real browsers, a stealth Chromium for pages that need JavaScript, a parser that keeps finding elements after a site's layout changes, Scrapy-style spiders, a command line, and an MCP server that gives Claude Code, Codex and other agents seventeen scraping tools.
On an exe.dev VM, that becomes a scraping box. The MCP server runs as a service, so every agent on the VM shares one server, and agents on your computer or another VM reach it through exe.dev's HTTPS proxy, private behind exe.dev login and tokens. The VM is always on, so spiders run on a timer and keep their results in SQLite on a persistent disk, and the agents use exe.dev's LLM integration, with no API key on the VM.
Once it's set up, an agent answers questions like this one by scraping:
$ claude -p "Use the scrapling make_request tool to fetch https://books.toscrape.com/catalogue/category/books/mystery_3/index.html with the CSS selector 'article.product_pod'. List the five cheapest books on that page as 'title - price', cheapest first."
These are the five cheapest books on that page, cheapest first:
1. Tastes Like Fear (DI Marnie Rome #3) - £10.69
2. Hide Away (Eve Duncan #20) - £11.84
3. Playing with Fire - £13.71
4. That Darkness (Gardiner and Renner #1) - £13.92
5. A Murder in Time - £16.64
What you get#
- Scrapling 0.5.1 with its browser installed, and
scrapling extractfetching pages with HTTP and with the stealth browser - Scrapling's MCP server on port 8765 as the
scrapling-mcpsystemd service, restarted after a reboot, with its token in a mode-600 file - Claude Code and Codex on the VM using exe.dev's LLM integration, both connected to the MCP server
- Scrapling's Agent Skill installed for Claude Code, so it writes spiders against the current API
- A Claude Code run that scrapes books.toscrape.com through the MCP server
What's running, and where
http://localhost:8765/mcp
the MCP server, for Claude Code, Codex and other agents on the VM
https://my-scrapling.exe.xyz:8765/mcp
the same server for agents elsewhere, with a VM token and the server's token
ssh my-scrapling.exe.xyz
your terminal, for scrapling extract and spiders
scrapling CLI and scrapling-mcp
Deploy with Shelley#
Click Deploy and Shelley will set up Scrapling on a new VM, then check that it works. No config-file wrangling required.
Prefer a terminal?
curl -fsSL \
https://exe-staging.dev/deploy/scrapling/prompt.txt \
| ssh exe.dev new --prompt=/dev/stdinWhat Shelley is told
Set up Scrapling on this VM by following the exe.dev recipe at https://exe-staging.dev/deploy/scrapling.md?via=deploy from top to bottom.
The recipe's example VM name is my-scrapling. This VM's name is the output of `hostname`: use it wherever the recipe writes my-scrapling. Skip the step that creates the VM; you are on it.
You are done when all of these are true. Check each one yourself:
- Scrapling 0.5.1 with its browser installed, and `scrapling extract` fetching pages with HTTP and with the stealth browser
- Scrapling's MCP server on port 8765 as the `scrapling-mcp` systemd service, restarted after a reboot, with its token in a mode-600 file
- Claude Code and Codex on the VM using exe.dev's LLM integration, both connected to the MCP server
- Scrapling's Agent Skill installed for Claude Code, so it writes spiders against the current API
- A Claude Code run that scrapes books.toscrape.com through the MCP server
Run the VM-side commands in "Set up by hand" only; skip creating the VM.
If `exeuntu configure` reports an existing user config, leave it alone,
and if `~/.codex/config.toml` already has `[mcp_servers.scrapling]`,
don't add it twice. Verify with the checks in that section. Don't set up
the timer, sign in to any site, or create VM tokens unless the person
asks. Finish by showing the Claude Code answer, and tell the owner how
to reach the server from their computer ("Use it from your computer or
another VM").
Finish with the URLs I should open.
Also at /deploy/scrapling/prompt.txt. The recipe it follows is /deploy/scrapling.md.
Set up the VM#
Deploy does this for you. By hand, it's a few blocks of commands: install Scrapling and its browser, start the MCP server as a service, and connect Claude Code and Codex to it.
Set up by hand
On your computer, create a VM and connect. Pick any free name; if you've already clicked Deploy or have a VM, skip the first command and use its name:
ssh exe.dev new --name=my-scrapling
ssh my-scrapling.exe.xyz
Install Scrapling#
On the VM, install Scrapling with all its extras as a
uv tool, then its browser and the system
libraries the browser needs. The Playwright pin keeps the browser that
scrapling install downloads in step with the one Scrapling drives (see
the FAQ):
uv tool install "scrapling[all]==0.5.1" --with "playwright==1.64.1"
export PATH="$HOME/.local/bin:$PATH"
scrapling install
The export is for this shell only; new logins find ~/.local/bin on
their own. scrapling install takes under a minute and uses sudo for
the libraries. Check both kinds of fetch: a plain HTTP request, and the
stealth browser on a page whose quotes only JavaScript renders:
mkdir -p ~/scrapes && cd ~/scrapes
scrapling extract get "https://quotes.toscrape.com/" quotes.md -s "div.quote"
scrapling extract fetch "https://quotes.toscrape.com/js/" js.md -s "span.text"
head -3 quotes.md js.md
Each command logs Fetched (200) and Content successfully saved, and
both files start with Einstein's quote about thinking.
Run the MCP server as a service#
The server's Streamable HTTP transport refuses to start without a token.
Generate one into a file only you can read, then start the server on port
8765 with systemd. --allowed-host turns on the MCP SDK's DNS-rebinding
protection for the names it's reached by; %H is the VM's name:
mkdir -p ~/scrapling/state && chmod 700 ~/scrapling
(umask 077; echo "SCRAPLING_MCP_AUTH_TOKEN=$(openssl rand -hex 32)" > ~/scrapling/mcp.env)
sudo tee /etc/systemd/system/scrapling-mcp.service >/dev/null <<'EOF'
[Unit]
Description=Scrapling MCP server
After=network-online.target
Wants=network-online.target
[Service]
User=exedev
WorkingDirectory=/home/exedev/scrapling
EnvironmentFile=/home/exedev/scrapling/mcp.env
ExecStart=/home/exedev/.local/bin/scrapling-mcp --http --host 0.0.0.0 --port 8765 --allowed-host %H.exe.xyz:8765 --allowed-host localhost:8765 --allowed-host 127.0.0.1:8765
Restart=on-failure
UMask=0077
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now scrapling-mcp
timeout 60 sh -c 'until curl -so /dev/null localhost:8765/; do sleep 1; done'
The last line waits the few seconds the server takes to start. Check it:
systemctl is-active scrapling-mcp prints active, and a request
without the token gets 401:
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8765/mcp -d '{}'
The server logs a MCPDeprecationWarning about
validate_token_resource at startup; it's harmless.
Connect Claude Code and Codex#
Connect both agents to the LLM integration, then add the server to each, with its token:
exeuntu configure claude
exeuntu configure codex
TOKEN=$(cut -d= -f2 ~/scrapling/mcp.env)
claude mcp add --scope user --transport http scrapling http://localhost:8765/mcp \
--header "Authorization: Bearer $TOKEN"
cat >> ~/.codex/config.toml <<EOF
[mcp_servers.scrapling]
url = "http://localhost:8765/mcp"
http_headers = { Authorization = "Bearer $TOKEN" }
EOF
Then install Scrapling's Agent Skill for Claude Code. It teaches the agent Scrapling's current API, so the spiders it writes work with 0.5:
mkdir -p ~/.claude/skills /tmp/skill
curl -fsSLo /tmp/skill.zip https://github.com/D4Vinci/Scrapling/raw/refs/tags/v0.5.1/agent-skill/Scrapling-Skill.zip
unzip -q /tmp/skill.zip -d /tmp/skill
rm -rf ~/.claude/skills/scrapling-official
mv /tmp/skill/Scrapling-Skill ~/.claude/skills/scrapling-official
rm -rf /tmp/skill /tmp/skill.zip
Check it end to end. claude mcp list shows
scrapling: http://localhost:8765/mcp (HTTP) - ✔ Connected, and Claude
Code answers with the five cheapest mystery books, starting with
Tastes Like Fear at £10.69:
mkdir -p ~/project && cd ~/project
claude mcp list
claude -p "Use the scrapling make_request tool to fetch https://books.toscrape.com/catalogue/category/books/mystery_3/index.html with the CSS selector 'article.product_pod'. List the five cheapest books on that page as 'title - price', cheapest first." --allowedTools "mcp__scrapling__make_request"
Ask an agent to scrape#
On the VM, start claude or codex in a project folder and ask in plain
words. Naming the tool saves the agent a guess, and a CSS selector keeps
the page small, which saves tokens:
Use the scrapling browser_fetch_once tool to open https://quotes.toscrape.com/js/ with the CSS selector 'div.quote', then tell me how many quotes are on the page and who wrote the first one.
The answer is ten quotes, the first by Albert Einstein; that page builds
its quotes with JavaScript, so it needs the browser. Codex works the same
way: codex exec --skip-git-repo-check "Use the scrapling MCP server's make_request tool to ...".
Which tools to ask for:
| Tool | Use it for |
|---|---|
make_request |
one page or API call over HTTP, with a real browser's TLS fingerprint and headers; fast, no browser |
browser_fetch_once |
one page that needs JavaScript |
open_request_session, session_make_request |
several HTTP requests that share cookies |
browser_open, browser_fetch, browser_actions |
a browser session the agent clicks, types and scrolls in, page by page |
browser_screenshot, browser_extract, browser_network_requests |
look at the open page, read part of it, or find the API calls behind it |
browser_save_state, browser_load_state |
keep a login (see below) |
Tell the agent to close sessions when it's done; open browser sessions
hold memory until it does (list_sessions shows them). Scrapling strips
hidden text, HTML comments and zero-width characters that pages use to
smuggle instructions to agents, as long as main_content_only stays on,
which is the default. That's a filter, not a guarantee: don't give an
agent that reads arbitrary pages more power than you'd give the pages.
Scrape from the terminal#
scrapling extract saves a page, or the parts matching a CSS selector, as
Markdown, HTML or text, chosen by the file's extension. get is a plain
HTTP request; fetch uses the stealth browser:
cd ~/scrapes
scrapling extract get "https://books.toscrape.com/" titles.txt -s "article.product_pod h3 a"
scrapling extract fetch "https://quotes.toscrape.com/js/" page.md --network-idle
scrapling extract get "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" book.md --ai-targeted
--ai-targeted keeps the main content and strips the same hidden text as
the MCP server: use it for anything a model will read. scrapling shell
opens an IPython console with Scrapling loaded, for trying selectors, and
scrapling extract fetch --help lists the browser's options.
Use it from your computer or another VM#
The server is also at https://my-scrapling.exe.xyz:8765/mcp, behind
exe.dev's proxy. A program reaching it needs two tokens: a
token for the VM, which gets it through the
proxy, and the server's own token, which Scrapling checks. On your
computer, create the VM token and copy the server's token, each into a
file only you can read:
(umask 077
ssh exe.dev ssh-key generate-api-key --vm=my-scrapling --label=scrapling-mcp > ~/.scrapling-vm-token
ssh my-scrapling.exe.xyz 'cut -d= -f2 ~/scrapling/mcp.env' > ~/.scrapling-mcp-token)
The first file should hold a token starting with exe1.. Then add the
server to Claude Code on your computer:
claude mcp add --scope user --transport http scrapling https://my-scrapling.exe.xyz:8765/mcp \
--header "X-Exedev-Authorization: Bearer $(cat ~/.scrapling-vm-token)" \
--header "Authorization: Bearer $(cat ~/.scrapling-mcp-token)"
claude mcp list
claude mcp list shows the server as ✔ Connected. Ask Claude Code to
scrape something, and the request runs on the VM: its IP address, its
browser, its saved logins. exe.dev's proxy consumes the
X-Exedev-Authorization header and passes Authorization on to
Scrapling. Without the VM token, the proxy answers 401 and nothing
reaches the VM. Other MCP clients take the same URL and two headers.
For an agent on another of your exe.dev VMs, you can keep both tokens off
that VM with a VM-to-VM integration: exe.dev
adds the VM token at the edge, and --bearer adds Scrapling's. On your
computer (my-agent-vm is the VM that runs the agent):
ssh exe.dev integrations add http-proxy --name scrapling \
--target https://my-scrapling.exe.xyz:8765/ --peer \
--bearer "$(cat ~/.scrapling-mcp-token)" --attach vm:my-agent-vm
On my-agent-vm, the server is then
claude mcp add --transport http scrapling https://scrapling.int.exe.xyz/mcp,
with no headers.
Warning
Whoever can call the server can make this VM fetch any URL, including services on the VM itself and the exe.dev integrations attached to it, such as the LLM integration. Hand out the tokens like you'd hand out a shell, give the VM only the integrations it needs, and rotate the server's token (below) if one leaks.
Write spiders and crawl on a schedule#
For more than a few pages, write a spider: a Python class that says where to start, what to pull out of each page, and which links to follow. Scrapling runs it with concurrency limits, delays, robots.txt checks, retries, and pause and resume. books.py is an example: it walks all 50 pages of books.toscrape.com, two requests at a time with half a second between them, and adds every book's title, price, rating and stock to a SQLite table, stamped with the time of the crawl. On the VM:
mkdir -p ~/scrapling/crawls && cd ~/scrapling/crawls
curl -fsSLO https://exe-staging.dev/deploy/scrapling/books.py
uv run books.py
The script pins scrapling[fetchers]==0.5.1 in its header, so uv run
builds its environment the first time. After about 35 seconds:
1000 books saved to /home/exedev/scrapling/data/books.db (50 requests, 34 s)
Query it with sqlite3:
sqlite3 -header -column ~/scrapling/data/books.db "
SELECT title, price, rating FROM prices
WHERE crawled_at = (SELECT max(crawled_at) FROM prices) AND in_stock
ORDER BY price LIMIT 5"
Run it every morning with a systemd timer. It runs whether or not you're connected, and catches up after a reboot:
sudo tee /etc/systemd/system/scrapling-books.service >/dev/null <<'EOF'
[Unit]
Description=Crawl books.toscrape.com with Scrapling
[Service]
Type=oneshot
User=exedev
WorkingDirectory=/home/exedev/scrapling/crawls
ExecStart=/usr/local/bin/uv run books.py
EOF
sudo tee /etc/systemd/system/scrapling-books.timer >/dev/null <<'EOF'
[Unit]
Description=Crawl books.toscrape.com every day
[Timer]
OnCalendar=*-*-* 06:00:00 UTC
RandomizedDelaySec=15m
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now scrapling-books.timer
sudo systemctl start scrapling-books runs it now,
journalctl -u scrapling-books -n 5 shows the summary line, and
systemctl list-timers shows the next run. Each run adds a new set of
rows, so you can compare crawls, for example prices that changed since
the one before:
sqlite3 -header -column ~/scrapling/data/books.db "
WITH runs AS (SELECT DISTINCT crawled_at AS t FROM prices ORDER BY t DESC LIMIT 2)
SELECT new.title, old.price AS was, new.price AS now
FROM prices new JOIN prices old USING (url)
WHERE new.crawled_at = (SELECT max(t) FROM runs)
AND old.crawled_at = (SELECT min(t) FROM runs)
AND new.price <> old.price"
(The practice site's prices never change, so this one prints nothing.) For a database that other programs read and write, see SQLite in production.
To write your own spider, ask Claude Code; the Agent Skill gives it Scrapling's current API:
Use the scrapling-official skill. Write a Scrapling spider, quotes.py, with uv inline script metadata pinning scrapling[fetchers]==0.5.1, that crawls every page of https://quotes.toscrape.com/ politely (obey robots.txt, a delay of 1 second) and writes each quote's text, author and tags to quotes.jsonl. Run it with
uv run quotes.py.
In testing it wrote a working spider on the first try and saved all 100
quotes. While you change a spider's parsing, set
development_mode = True (the commented line in books.py): the first
run saves every response under .scrapling_cache/<spider name>/, and
later runs replay them from disk without touching the site. A rerun of
books.py drops from 35 seconds to 3. Turn it off before the timer uses
the spider: cached pages never expire.
Extract fields with a model#
Selectors are exact and free, so use them when a page has structure. When
it doesn't, or you want a summary, hand the cleaned page to a model
through the LLM integration. Scrapling has no model calls of its own; this
is jq and curl with a JSON schema, so the answer is always valid JSON:
cd ~/scrapes
scrapling extract get "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" book.md --ai-targeted
jq -n --rawfile page book.md '{
model: "gpt-6-luna",
instructions: "Extract the book from this page.",
input: $page,
text: {format: {type: "json_schema", name: "book", strict: true, schema: {
type: "object", additionalProperties: false,
required: ["title", "price_gbp", "upc", "copies_in_stock", "summary"],
properties: {
title: {type: "string"}, price_gbp: {type: "number"}, upc: {type: "string"},
copies_in_stock: {type: "integer"}, summary: {type: "string"}}}}}
}' | curl -s https://llm.int.exe.xyz/v1/responses -H 'Content-Type: application/json' -d @- \
| jq -r '.output[] | select(.type == "message") | .content[0].text' | jq .
{
"title": "A Light in the Attic",
"price_gbp": 51.77,
"upc": "a897fe39b1053632",
"copies_in_stock": 22,
"summary": "A special 20th-anniversary edition of Shel Silverstein’s classic collection of humorous poetry and drawings."
}
No API key: the integration supplies it and bills your exe.dev account.
In a spider, do the same per item with any OpenAI-compatible client
pointed at https://llm.int.exe.xyz/v1.
Sites that need a login#
The MCP server can save a browser session's cookies and local storage to
a file on the VM and load them into a later session, so an agent signs in
once. Paths are relative to ~/scrapling, the service's folder. The
practice site accepts any name and password:
With the scrapling tools: open a browser session, fetch https://quotes.toscrape.com/login, fill the username and password fields with 'demo', submit the form, save the browser state to state/quotes.json and close the session. Then open a new session, load state/quotes.json, fetch https://quotes.toscrape.com/ and tell me whether the page shows a Logout link.
It does, and ~/scrapling/state/quotes.json holds the session cookie. In
your own spiders, StealthySession takes the same files.
Warning
A state file is a logged-in session: anyone who can read it, or call the MCP server, can act as that account until the site logs it out. Keep them in
~/scrapling/state(the service creates files only you can read), use accounts made for scraping with as few rights as possible, never your email, bank or admin accounts, and delete files you no longer need. Passwords for scripts belong in files with mode 600 or in agent secrets, not in prompts.
When you need a visible browser#
Scrapling's browser runs headless, which is what scraping wants: no desktop, little memory (the idle server uses about 170 MB). Use a browser you can see when a person has to step in: to sign in with two-factor authentication, to solve a check by hand, or to watch an agent work out a site. Two ways:
- Cloud browser runs Chrome you open in a tab
and exposes it over the DevTools protocol at
http://localhost:9222. Ask the agent to open a browser session connected over CDP to http://localhost:9222, and Scrapling drives that Chrome, with the logins you made in it by hand. - Browser Use puts an AI agent in a Chrome on a Linux desktop, for tasks described in words rather than scraped.
Scrape responsibly#
Follow each site's terms and robots.txt (robots_txt_obey = True), crawl
slowly, leave personal data alone, and don't use stealth or
challenge-solving where a site has said no to bots: what the VM fetches is
on you, under exe.dev's terms of service. Test
against books.toscrape.com and quotes.toscrape.com, which exist for scraping.
Back up and restore
What's worth keeping is in ~/scrapling: crawls/ (your spiders),
data/ (results), state/ (saved logins) and mcp.env (the server's
token). Copying the VM keeps everything, services included. On your
computer:
ssh exe.dev cp my-scrapling my-scrapling-backup
For a copy of just the data, back up the database while nothing writes to it, and copy the folder to your computer:
sqlite3 ~/scrapling/data/books.db ".backup '/home/exedev/scrapling/data/books-backup.db'"
scp -r my-scrapling.exe.xyz:scrapling ./scrapling-backup
To restore on a VM set up the same way, copy the folder back with
scp -r ./scrapling-backup my-scrapling.exe.xyz:scrapling and run
sudo systemctl restart scrapling-mcp. The copy holds saved logins and
the token: keep it private.
Update
Scrapling releases often, sometimes with breaking changes (0.5 renamed
MCP tools and removed the bulk ones). Read the changelog
first, then on the VM install the version you want. Keep Playwright on
the minor version of the Patchright that Scrapling installs (Patchright
1.64.4 goes with Playwright 1.64.1); this shows the installed pair:
uv pip list --python ~/.local/share/uv/tools/scrapling/bin/python | grep wright.
uv tool install --force "scrapling[all]==0.5.1" --with "playwright==1.64.1"
scrapling install --force
sudo systemctl restart scrapling-mcp
Change the pin in your spiders' headers too, and update the Agent Skill
by repeating its commands with the new tag in the URL. To rotate the
server's token, write a new mcp.env as in setup, restart the service,
and add the server to your agents again (claude mcp remove scrapling -s user
first, and replace the line in ~/.codex/config.toml).
Troubleshooting
Executable doesn't exist at ~/.cache/ms-playwright/chromium-…. Playwright and Patchright want different Chromium builds. Reinstall with the--with "playwright==…"pin from setup, thenscrapling install --force.scrapling: command not found.~/.local/binisn't on this shell'sPATHyet: log in again, orexport PATH="$HOME/.local/bin:$PATH".- The service won't start.
journalctl -u scrapling-mcp -n 50says why. "Refusing to serve the MCP server over HTTP without authentication" means~/scrapling/mcp.envis missing or empty. 401from the server. Without the VM token, it's exe.dev's proxy: the token is missing, expired, or for another VM (copies need their own). With it, the server's token is wrong or missing.421 Misdirected Request. The host name isn't one of the--allowed-hostvalues, for example after renaming the VM. Restart the service so%Hpicks up the new name.claude mcp listsays Failed to connect. Checksystemctl status scrapling-mcp, then that the header carries the current token.- A site answers
403,429or a challenge page. It's telling you to stop or slow down; see "Scrape responsibly". Some sites block cloud addresses outright. - Memory grows. Open browser sessions stay open until closed. Ask the
agent to
list_sessionsand close them, orsudo systemctl restart scrapling-mcp.
Frequently asked questions#
Is Scrapling free? What does this cost?
Scrapling is free and open source (BSD-3-Clause). You pay for the VM as part of your exe.dev plan. Agents and the extraction example use exe.dev's LLM integration, billed to your exe.dev account; exe.dev keeps a list of models and prices. Scrapling itself calls no model.
Does the Scrapling MCP server use stdio or HTTP?
Both. scrapling-mcp speaks stdio by default, which is enough for one agent on the same machine (claude mcp add scrapling -- scrapling-mcp). With --http it serves Streamable HTTP at /mcp and requires a bearer token. This page runs the HTTP server as a service, so every agent on the VM, and agents elsewhere with a VM token, share one server, its browser sessions and its saved logins.
Who can reach the MCP server?
Programs on the VM that have the server's token, and, through exe.dev's HTTPS proxy, only callers that are signed in to exe.dev with access to the VM or hold a token for the VM, and also send the server's token. Ports other than the VM's main one can't be made public. Anyone who gets through can make the VM fetch any URL, so treat both tokens like a shell on the VM.
Why the pinned Playwright version?
scrapling install downloads Chromium with Playwright, but Scrapling's browser runs on Patchright, a patched Playwright that trails it by a release. Since Playwright 1.65 (November 2026), a plain install gets a Chromium that Patchright can't find, and every browser fetch fails with "Executable doesn't exist". Pinning Playwright 1.64.1, the version Patchright 1.64.4 matches, makes both use the same Chromium.
Can I use the Docker image instead?
Yes: pyd4vinci/scrapling (and ghcr.io/d4vinci/scrapling) has the CLI, the browser and the MCP server. The uv install here is smaller to update, puts saved logins and spiders in plain folders, and lets spiders on the VM import Scrapling directly.
Can Scrapling get past Cloudflare and CAPTCHAs?
Its stealth browser has options to solve some Cloudflare challenges (--solve-cloudflare, solve_cloudflare), and the MCP server can work through some interactive checks. A challenge is a site telling you it doesn't want automated traffic: check its terms first, and don't use these options where you aren't allowed to scrape.
Published Nov 19, 2026 · Updated Nov 19, 2026 · tested end to end Nov 19, 2026 · Markdown · manifest.json · Something wrong? Tell us