Input
URL
https://example.com/article
01 · product
One URL in.
Clean JSON or Markdown out.
A resilient URL reader for agent pipelines — articles, PDFs, images, and hostile media routed through HTTP, SEO, CloakBrowser, and archive fallbacks.
02 · proof
URL in. Structured reading out.
Same contract for CLI one-shots and the local JSON API — title, summary, fetch route, warnings, optional cleaned body.
Output
JSON · Markdown
"status": "ok", "content_type": "article", "title": "Example Domain Article", "summary": "A short readable extract…", "fetch_method": "http", "warnings": []
JSON
Markdown
optional content body
warnings, never silent
03 · formats
Public routes, explicit limits
Specialized extractors for common public page shapes. Authenticated or gated pages stay partial or error — never silently claimed as full reads.
Articles
HTML articles via OpenGraph, JSON-LD, and readability-style extractors into title, summary, and optional cleaned content.
PDFs
Text extraction from public PDF URLs when optional PDF extras are installed.
Images
Image and vision metadata with deterministic placeholders when no vision model is configured.
YouTube
Public video metadata plus public transcripts when available — no private account session required.
Public profile, company, job, Pulse, and post pages. Authenticated or gated LinkedIn views remain partial or error.
Cloak-first public Reddit reads via the optional browser extra. Login, CAPTCHA, and consent gates stay partial or error — never automated.
Login walls, cookie walls, CAPTCHA, and subscriber-only bodies are reported through status and warnings — not faked as complete reads.
04 · pipeline
Layered fetch, not one browser for everything
Most public pages resolve over direct HTTP. CloakBrowser is an optional later stage — not the default path for every URL.
01
Direct HTTP
Timeouts and size caps on outbound fetches.
02
Extractors
Article, PDF, image, and media routes.
03
SEO variants
Bot user-agent and referer fallbacks when useful.
04
CloakBrowseroptional
Optional render for hostile media.
05
Archives
Public cache snapshots as a last resort.
06
Jina LinkedInopt-in
Opt-in helper for public LinkedIn reads.
Hostile pages need layers — not a single forever-browser.
Local-first trust05 · local-first
Local-first trust boundary
The CLI and local service are meant to run on your machine — not as an open internet proxy.
Local-first by default
The CLI and local service can read internal URLs when you run them deliberately. The HTTP service listens on localhost by default — do not expose it as a public proxy without network policy and authentication.
Route-level blocking
External fetch routes, social extractors, and the Jina LinkedIn helper apply their own private, loopback, and link-local blocking — independent of the core reader.
Size and timeouts
Hard max-size and request timeouts keep runaway downloads and hung sockets from stalling agent jobs.
No cookies by default
No login flow ships by default. Operator-owned browser profiles are explicit opt-in and stay outside the package.
06 · cli + api
CLI, JSON API, extensible routing
One reader contract across one-shot CLI runs and a small local HTTP service.
CLI
One-shot reads with JSON or Markdown output, optional content body, and per-run fallback toggles.
JSON API
Local service with /health, /summarize, /read, /markdown, and dependency-free /openapi.json.
Routing cache
Metadata-only per-domain strategy cache stores fetch routes — never cookies, tokens, or page bodies.
$ supersocks-url-scraper https://example.com/article { "status": "ok", "url": "https://example.com/article", "content_type": "article", "title": "Example Domain Article", "summary": "A short readable extract…", "length": 420, "fetch_method": "http", "warnings": [] }
One command, one JSON contract — the same shape the local service returns.
07 · example
Example JSON response
Realistic public-page shape. No private hosts, tokens, or account data.
{
"status": "ok",
"url": "https://example.com/article",
"content_type": "article",
"title": "How public URL readers feed agent pipelines",
"summary": "A public HTTP page is fetched, cleaned into readable text, and returned as a small JSON contract with title, summary, fetch route, and warnings.",
"length": 812,
"fetch_method": "http",
"warnings": [],
"content": "Optional cleaned body when --include-content is set."
}
Pick the extras you need, copy the pipx line, run local.
Configure install08 · install
Run it locally
Install from GitHub with pipx. Pick the Python extras you need — the command below updates as you toggle.
Web core is always included. Public LinkedIn guest pages work in the core without an extra.
Optional extras match those published on
supersocks-url-scraper
main: article, pdf, browser, and youtube.
Reddit, Instagram, and Facebook are Cloak-first and require the browser extra.
Prerequisites
-
X / twitter-cli:
provide
TWITTER_AUTH_TOKENandTWITTER_CT0explicitly in your own environment. This page never asks for or stores those values. -
Reddit / Instagram / Facebook (Cloak-first):
requires the
browserextra. Runs headless by default. An optional operator-owned persistent profile (BROWSER_PROFILE_DIR/SOCIAL_BROWSER_PROFILE_DIR) can keep a warmed session; cookies stay in that directory only. For login, MFA, CAPTCHA, or consent, warm the profile yourself under an existing VNC / Xvfb / display (CLOAK_HEADLESS=0) — this package never automates credentials or cookies. -
OpenCLI desktop fallback (opt-in only):
not installed by this command. After Cloak-first Instagram/Facebook, set
SOCIAL_OPENCLI_FALLBACK=1yourself if you already run OpenCLI against a Chrome session you control. Never auto-installed; never ZIP; never default.
Install commands
pipx install 'supersocks-url-scraper[article,pdf,browser,youtube] @ git+https://github.com/iamsupersocks/supersocks-url-scraper.git'
Always starts with GitHub + pipx for the scraper extras (article, pdf, browser, youtube). X may append pipx install twitter-cli. No ZIP installs and no automatic npm OpenCLI line.
Inspiration:
Agent Reach.
Upstream:
twitter-cli,
CloakBrowser (browser extra) for Reddit / Instagram / Facebook,
and optional desktop
OpenCLI
only when you set SOCIAL_OPENCLI_FALLBACK=1.
GitHub
pipx from GitHub
09 · limits · faq
Honest limits, clear answers
Public pages only. Gates stay gates. The reader reports what it could not finish.
What pages can it read?
Public articles, PDFs (with the pdf extra), images, YouTube metadata/transcripts when public, LinkedIn guest pages, and Cloak-first public Reddit via the browser extra. Authenticated or gated views stay partial or error.
What about login walls and CAPTCHA?
Login walls, cookie walls, CAPTCHA, and subscriber-only bodies are reported through status and warnings — never faked as complete reads. The package does not automate credentials or cookies.
Is the HTTP service safe to expose?
No. The CLI and local service are local-first. The HTTP service listens on localhost by default — do not expose it as a public proxy without network policy and authentication.
Does it store cookies or tokens?
No cookies by default. The routing cache stores metadata-only per-domain fetch strategies — never cookies, tokens, or page bodies. Operator-owned browser profiles are explicit opt-in and stay outside the package.
Which install extras exist?
Published extras on main: article, pdf, browser, and youtube. Web core (including public LinkedIn guest extraction) is always included. Reddit, Instagram, and Facebook require browser.
10 · ship
Read the hostile web on your machine.
Install from GitHub with pipx, keep the trust boundary local, feed agents a clean JSON or Markdown contract.