hyperresearch v0.11.0：为抓取流程加入 SSRF 防护与响应大小上限

Official release notes (https://github.com/jordan-gibbs/hyperresearch/releases/tag/v0.11.0) ### SSRF gate + size caps for fetches (builtin, crawl4ai entry points, PDF and image downloads) - **The builtin provider, the crawl4ai entry points, and PDF and image downloads now go through an SSRF gate** (`web/safe_http.py`): http/https only; hostnames resolving to private, loopback, link-local, multicast, reserved, CGNAT (RFC 6598), or unspecified addresses are refused (cloud metadata, RFC1918, `[::1]`, `100.64.0.0/10`, …). Not covered: the fixed-host API calls in `core/embed.py` and `core/scholar.py`, which take no attacker-influenced URLs. On the `safe_get` paths (builtin provider, PDF and image downloads) redirects are followed manually and every hop is revalidated. The browser lanes (crawl4ai `fetch()`, `fetch_many()`, and the visible-window path) are gated at entry AND re-checked on the URL the browser actually landed on — the browser follows redirects internally, so the entry check alone cannot vouch for the destination. A post-hoc recheck cannot stop the request that already fired (blind SSRF survives it), but it keeps private-network content out of the vault; the same-host case skips the extra DNS lookup. Refused URLs in a batch are logged and skipped, not fatal. The hostname check is best-effort against DNS rebinding (httpx re-resolves at connect time); the residual window is documented in the module docstring. - **`[fetch] allow_private_hosts`** — escape hatch for self-hosted mirrors and intranet sources: hostnames (exact, case-insensitive) or CIDRs that may be fetched despite resolving to private space. Empty by default; consulted by `check_url` everywhere, including redirect hops and the final-URL recheck. A malformed CIDR entry errors loudly instead of being silently ignored. - **Response-size caps**, streamed and enforced mid-body so a lying or chunked server cannot exhaust memory. Configurable via new `[fetch]` settings `max_html_bytes` (10 MiB), `max_pdf_bytes` (25 MiB), `max_image_bytes` (2 MiB). A gate refusal on an image download is printed, not silently swallowed. - **No unverified-TLS retry.** A certificate failure on a PDF fetch is a refusal that names the existing `pdf_verify_tls = false` opt-out — never an automatic verify-off retry, which a MITM could force with a bad cert. The refusal is its own exception type (`CertVerificationError`), and a cert-refused PDF is never handed to the browser fallback lane: that lane runs with TLS errors ignored, so "falling back" there would be the automatic unverified retry by another name. In a batch it is a loud skip instead. ### Scholarly discovery: eight sources through one client layer Academic discovery used to be four URL templates rendered into the agent's instructions by `core/agent_docs.py`, which the model was trusted to assemble and call by hand. No retry, no rate limiting, no dedup, no offline tests — and one of those templates shipped `mailto=research@example.com`, a shared placeholder on every install, which is exactly the anti-pattern the open-access resolver refuses to commit for Unpaywall. It is now a real package. - **`hpr scholar search` and `hpr scholar sources`.** One query hits every configured provider, merges records that are the same work, and returns one list. `sources` lists what is wired, what each covers, and why anything is unavailable — so a user is never guessing which source to reach for. - **Dedup is by DOI first, then normalized title within ±1 year.** The year tolerance is deliberate: providers disagree systematically about online-first versus print year for the same article. Two *different* DOIs never merge regardless of title, which is what stops four 2025 reprints of a famous paper from collapsing into one record. A work confirmed by several providers carries them in `also_in`, with the highest citation count and the longest abstract. - **`--limit` is per provider, not a cap on the merged list.** A post-merge cap would show only the first provider's records at small limits and make every other source look empty. - **Providers: OpenAlex, Crossref, CORE, DOAB, ClinicalTrials.gov, SEC EDGAR, FRED.** Each is a real client through one cache-first, rate-limited HTTP seam (`scholar/base.py`), with fixtures matching live response shapes. OpenAlex's abstracts arrive as an inverted word-position index and are reconstructed; Crossref's arrive as JATS XML and are stripped; ClinicalTrials.gov and EDGAR were verified against the live services, including EDGAR's User-Agent gate. - **Non-STEM coverage is a first-class goal.** OpenAlex and DOAB return books and book chapters, which matters because in the humanities the book is the unit of publication and nothing in the stack could find one before. DOAB is the only source that finds *the book* rather than a review of it. - **RePEc is listed as unavailable, on purpose.** Their API documents that it has no search function. Shipping an honest "cannot search" with a pointer to OpenAlex for the DOI-bearing series beats silently omitting the field. - **`HYPERRESEARCH_CONTACT_EMAIL` replaces the placeholder.** Set it and OpenAlex and Crossref serve you from their polite pools, and SEC EDGAR — which rejects any request without a contact address — becomes available. Unset, no address is sent at all. - **Specialist records are tagged, not disguised.** Trials, filings and economic series come back with `work_type` set so the pipeline never treats a 10-K as a paper. - **`FRED_API_KEY` is never cached.** FRED authenticates by query parameter and the cache keys on URL, so FRED requests bypass the cache rather than write the key into the vault's SQLite in plaintext. ### CORE is the third open-access resolver Recovery used to ask Unpaywall, then Europe PMC. But `contact_email` is empty by default, which disables Unpaywall, and Europe PMC is biomedical only — so a stock install's open-access recovery covered almost nothing outside biomedicine while the 0.10.0 notes presented it as a headline feature. [CORE](https://core.ac.uk/) now runs third: it is the largest full-text open-access aggregator, and unlike Unpaywall it hosts the plain text directly rather than pointing at a repository that may 403. - Activates when `CORE_API_KEY` is set; skipped silently otherwise, the same way Unpaywall is skipped without a contact address. - Every existing invariant holds and is tested: a candidate is accepted only if it is longer than what we had and clears `oa_min_full_text_chars`; failure is soft; resolver URLs go through `check_oa_url`; `oa_max_attempts` is honoured; the four-place disclosure contract is populated; `oa_source` gains the value `core`. - **Version is recorded honestly.** CORE does not reliably say which version it holds, so `oa_version` stays unset unless CORE marks the record a preprint — and the banner then says the version is unrecorded and tells the reader to quote with care, rather than implying version of record. ### Agent prose now points at the client The "Academic APIs before web search" section of the injected agent instructions tells agents to run `hpr scholar search` and not to hand-assemble API URLs, and explains what each source is for. ### Contributed fixes - **`run finish` no longer blocks every light-classified run started on the installed gear (@maximilliangrand in #95).** `verify_run()` took its required-artifact step set from the manifest's `profile_steps`, so a run initialized with `--profile full` whose step-1 decomposition classified it `light` was asked for `critic-findings-*.json` and `patch-log.json` — artifacts the light tier correctly never writes, because steps 12 and 14 are skipped by the tier gate. The run did all its light-tier work and then sat at `blocked (verify)` with no legitimate way to pass. The gate now resolves the tier declared in `prompt-decomposition.json` when it disagrees with the manifest profile, which is what the router already documents ("the manifest's profile field is informational — the decomposition's tier rules"). A missing, unreadable or unknown tier still falls back to `profile_steps`. - **The MCP `fetch_url` tool works again, and a deleted note's URL can be fetched again (#84, fixed by @AmirF194 in #86).** Two bugs on the same path. `Crawl4AIProvider.fetch()` and `fetch_many()` called `asyncio.run()`, which raises before the coroutine runs when the caller already has a loop — and the MCP server dispatches sync tools on its own loop thread, so the tool failed on every call while the CLI never noticed. Both now go through a wrapper that runs the coroutine on a dedicated thread when a loop is present. Separately, `sources.note_id` is `ON DELETE SET NULL`, so deleting a note leaves its row behind with a NULL id; every duplicate-URL check tested row truthiness, so that URL was `DUPLICATE_URL` ("already fetched as note 'None'") forever. All four fetch paths (`fetch_and_save`, `hpr fetch`, `hpr fetch-batch`, `hpr research`) now share one orphan-aware check. The three CLI paths also record the source with `INSERT OR IGNORE`, which was a no-op on the orphaned row too — so the new note was left with no source record, and `hpr fetch`'s duplicate-race detector then deleted the note it had just written and reported `note_id: null`. Those paths now reclaim the orphaned row after the insert; the guard on `note_id IS NULL` leaves a genuine race winner untouched. - **The PreToolUse reminder now reaches the model (#94; @dajiaohuang in #97 and #98).** The installed hook wrote its reminder to stderr and exited 0, and for PreToolUse that channel is discarded — so the "check the research base before searching the web" nudge shipped since 0.x had never once been delivered. It is emitted as `hookSpecificOutput.additionalContext` now, which is the documented injection channel. The matcher is narrowed to `WebSearch|WebFetch`: on `Glob` and `Grep` the advice is noise, and delivering it there for the first time would have made every local vault operation pay for a reminder about the web. - **Parallel Search is a fifth web provider (@georgeatparallel in #57).** `[web] provider = "parallel"` searches through Parallel's keyless Search MCP endpoint — the first search-capable provider that needs no account, since `builtin` cannot search at all and `exa` and `tavily` both need a key. Search only: bulk fetch waves degrade to per-URL, so it complements the crawl4ai fetch path rather than replacing it. The README gained a **Web providers** section listing all five on equal footing; none had been documented there before. This PR had been red on CI since August for an `asyncio.run()`-inside-a-running-loop error — the same class of bug #86 fixed — and got the same loop-safe runner. Repository README (current repository documentation, not release changes) The Most Powerful Deep Research Harness --- **Hyperresearch turns Claude Code into a deep research agent: one that currently leads the DeepResearch-Bench RACE leaderboard (benchmarked internally).** A tier-adaptive 16-step pipeline takes one prompt and produces an adversarially-audited report with full source provenance. Every source it reads lands in a persistent, searchable vault, so each session starts smarter than the last. Forward-looking projection from a stratified pilot against the DeepResearch-Bench leaderboard snapshot (https://huggingface.co/spaces/muset-ai/DeepResearch-Bench-Leaderboard). Third party validation is pending. ## Why it wins - **250+ sources in a single run.** The `premier` scale profile targets 100–130 in the width sweep alone; citation chasing and gap-fill fetches more than double what actually lands in the corpus. - **Every citation is verified before the report ships.** A skeptical cite-checker audits whether each cited source actually supports its sentence. Hallucinated quotes and unacknowledged retractions are hard blocks at the gate. - **Syndication doesn't count as consensus.** An independence audit clusters derivative copies, so five reprints of one press release argue with the weight of one source. - **Adversarial by construction.** Four critics attack every draft in parallel, and a tool-locked patcher can only apply surgical edits. It physically cannot rewrite the report. - **Eight scholarly sources, one query.** `hpr scholar search` hits OpenAlex, Crossref, CORE, DOAB, ClinicalTrials.gov, SEC EDGAR and FRED through one client layer and returns a single list deduplicated by DOI and title. Books, trials and filings come back alongside papers, each tagged so the pipeline knows which is which. The humanities and social sciences are covered on purpose, not as an afterthought. - **Paywalled papers get read, not skimmed.** A closed paper normally enters a vault as a 1,500-character abstract that the report then cites as though it had been read. Hyperresearch asks Unpaywall, Europe PMC and CORE for a legal open-access copy and stores the full text instead, even when the publisher blocks the fetch outright. Every substitution is disclosed in the note, the frontmatter, and the CLI output. - **Nothing is thrown away.** Every source lands in a searchable markdown-plus-SQLite vault that your next session reuses before it fetches anything new. - **Crashed runs resume.** Each run keeps a manifest; `run resume` picks up at the exact step where it died. - **Scales from 30 minutes to a dissertation.** Bounded queries auto-route to a 5-step fast path. Opt-in dissertation runs write 25K–80K words across chapters, from 300–450 sources. ## Install ```bash cd your-project pip install hyperresearch && hyperresearch install ``` Then `/hyperresearch ` in Claude Code. > Python 3.11–3.13. (3.14 not yet supported. Use `pyenv install 3.13`, `uv venv -p 3.13`, or `py -3.13 -m venv .venv`.) > > Power users: `hyperresearch install --global` makes `/hyperresearch` reachable from every Claude Code session anywhere, at the cost of ~15 lines in every session's system reminder. Per-project install (above) keeps unrelated CC sessions clean. --- ## The 16-step research pipeline The entry skill is a thin router. It pins down the canonical research query, then invokes one step skill per phase via Claude Code's `Skill` tool. Each step's procedure loads into context only when that step actually runs. That's what stops a long pipeline from quietly dropping steps as its context rots. | # | Step | What it does | Tiers | |---|---|---|---| | 1 | Decompose | Canonical query → atomic items + coverage matrix + tier classification | all | | 1.5 | Chapter partition | Group atomic items into 4–10 chapters; steps 2–10 then loop per chapter | dissertation | | 2 | Width sweep | Multi-perspective search plan + parallel fetcher waves | all | | 3 | Contradiction graph | Pair contradictions across the corpus into ranked clusters | full | | 4 | Loci analysis | Two parallel loci-analysts → scored loci with source budgets | full | | 5 | Depth investigation | K parallel depth-investigators → interim notes with committed positions | full | | 6 | Cross-locus reconcile | Reconcile committed positions → comparisons.md | full | | 7 | Source tensions | Extract expert disagreements → source-tensions.json | full | | 8 | Corpus critic | "What source would overturn this?" + targeted gap-fill fetch | full | | 9 | Evidence digest | Top claims + verbatim quotes → evidence-digest.md | full | | 10 | Triple draft | Per-angle source curation + 3 parallel draft sub-orchestrators (light: single draft) | all | | 11 | Synthesize | Plan + outline + spawn synthesizer subagent → final_report.md | full | | 12 | Critics | 4 adversarial critics in parallel → findings JSONs | full | | 13 | Gap-fetch | Targeted fetch wave for critic-identified vault gaps | full | | 14 | Patcher | Surgical Edit hunks applied to draft (tool-locked Read+Edit) | full | | 14.5 | Cite-check | Verify citation-sentence bindings; skeptical LLM spot-check; second surgical patch pass | full | | 15 | Polish | Hygiene + filler pass (tool-locked Read+Edit subagent) | all | | 16 | Readability audit | Recommender writes JSON suggestions; orchestrator selectively applies | all | ### Tiers and gears: the two scale levers **Tiers** route per query. Step 1 auto-classifies `light` vs `full`. `dissertation` is opt-in only; ask for it in your prompt. | Tier | What runs | Typical time | |---|---|---| | `light` | bounded factual queries, surveys, comparisons: 1 → 2 → 10 → 15 → 16 | ~30–40 min | | `full` (default) | deep argumentative analysis with adversarial review: all 16 steps + cite-check | ~1.5–2.5 h at `full` gear | | `dissertation` | chaptered mega-runs: 300–450 sources across 4–10 chapters, 25K–80K words | ~4–8 hours | **Gears** set the scale of the standard pipeline: the source targets, depth budgets, and word targets rendered into the step skills. ```bash hyperresearch profile list # all profiles + descriptions + current gear hyperresearch profile use premier # 100–130 sources, doubled depth budget (~3–5 h) hyperresearch profile use full # back to the 55–80-source baseline ``` The gear persists per project and survives reinstalls. Custom gears: define `[profile.]` in `.hyperresearch/config.toml` (any knob: source targets, loci caps, draft counts, word targets, per-agent models) and `profile use `. ### Run levers: what voice the report is written in Tiers and gears set how much work happens. Levers set what kind of report comes out, and step 1 picks them from your prompt's verb shape. An explicit directive in your prompt always wins. | Lever | Values | What changes | |---|---|---| | `register` | `teach` / `survey` / `analyze` / `advocate` | "Teach me X" gets a pedagogical explainer; "what's the landscape" gets a map of the field with no verdict; `analyze` (the default) gets the evaluative argument; `advocate` defends one named thesis | | `domain_notes` | freeform | Sourcing strategy, evidence norms, recency window for the field in question | | `inference_depth` | `surface` / `standard` / `deep` | The rabbithole dial. Step 4 can upgrade it after seeing what the corpus actually holds | The levers render into role-scoped shim files that spawn templates paste verbatim into subagent prompts, so the critics move with the register instead of undoing it. In `survey` register the dialectic critic flags unfair representation rather than missing commitment, and the polish auditor stops striking hedges. In `advocate` all of them tighten instead. The cite-checker and the ship gate receive no shim at all. Verification never softens by mode. ```bash hyperresearch levers set inference_depth=deep --rerender # go deeper mid-run hyperresearch run status -j # see what step 1 chose ``` ### The two load-bearing principles 1. **Patch, never regenerate.** After step 11 produces the synthesized report (or step 10 for light tier), the only modifications are surgical Edit hunks. The patcher and polish auditor are tool-locked to `[Read, Edit]` at the Claude Code allowlist level so they physically cannot Write a new draft. Per-hunk caps make "just rewrite it" mechanically impossible. Critic findings that don't fit a small hunk escalate as structural issues. 2. **Canonical research query is gospel.** The verbatim user prompt is persisted to `research/runs//query.md` once and re-read by every subsequent step and every spawned subagent. Wrapper requirements (save paths, citation format, terminal sections) are a separate contract. ### Subagent roster Models are profile config, not hardcode. The table shows the shipped defaults, and you can override any of them in `.hyperresearch/config.toml`: `[profile.full]` with `models = { fetcher = "haiku" }` swaps every fetcher to Haiku on the next install or `profile use`. | Agent | Default model | Role | |---|---|---| | `hyperresearch-fetcher` | Sonnet | URL fetching via crawl4ai; runs 8–12 in parallel per wave | | `hyperresearch-source-analyst` | Sonnet | End-to-end digest of any single long source >5000 words | | `hyperresearch-loci-analyst` | Sonnet | Reads the width corpus, returns 1–8 depth loci with rationale | | `hyperresearch-depth-investigator` | Sonnet | Investigates one locus, writes one interim note with a committed position | | `hyperresearch-corpus-critic` | Sonnet | "What source would overturn the current direction?" pre-draft gap analysis | | `hyperresearch-draft-orchestrator` | Opus | One per draft angle; reads its curated source list and writes one draft | | `hyperresearch-synthesizer` | Opus | Reads all 3 drafts, writes the final report (two-pass write, Read+Write locked) | | `hyperresearch-dialectic-critic` | Opus | Counter-evidence the draft missed | | `hyperresearch-depth-critic` | Opus | Shallow spots interim notes could fill | | `hyperresearch-width-critic` | Opus | Topical corners the corpus supports but the draft ignores | | `hyperresearch-instruction-critic` | Opus | Structural mismatches against the prompt's atomic items | | `hyperresearch-patcher` | Opus | Tool-locked `[Read, Edit]`. Applies critic findings as surgical Edit hunks | | `hyperresearch-cite-checker` | Sonnet | Skeptically verifies sampled citation-sentence bindings before ship | | `hyperresearch-polish-auditor` | Opus | Tool-locked `[Read, Edit]`. Cuts filler, strips hygiene leaks | | `hyperresearch-readability-recommender` | Opus | Writes JSON suggestions for paragraph rhythm and list/table conversion | | `hyperresearch-browser-fetcher` | Sonnet | Drains the escalation queue by driving your real Chrome (Claude-in-Chrome) | --- ## The vault: persistent, searchable, compounding Most deep research harnesses are one-shot: report out, everything else discarded. Hyperresearch keeps what it reads. Every fetched source lands in a SQLite-indexed vault that future sessions search before they fetch. ```bash hyperresearch search "ion-trap gate fidelity" -j # Full-text search hyperresearch search "quantum" --include-body -j # Full-body search hyperresearch note show -j # Batch-read notes hyperresearch graph hubs -j # Most-connected notes hyperresearch graph backlinks -j # Reverse links hyperresearch lint -j # Health check (broken links, missing tags) ``` **Markdown is truth, SQLite is cache.** Notes live as plain markdown with YAML frontmatter in `research/notes/`. The SQLite index is fully rebuildable: delete it and `hyperresearch sync` reconstructs it from the markdown. Open the vault in any editor, version it in git. You don't need the tool installed to read your own research. **PDFs fetch directly.** `hyperresearch fetch` auto-detects PDF URLs (arXiv, NBER, SSRN, direct `.pdf` links) and extracts full text via pymupdf. Raw PDFs land in `research/raw/.pdf` and the note's `raw_file:` frontmatter links back. **Provenance breadcrumbs.** Every fetched source carries a `--suggested-by` link back to whatever surfaced it. The chain forms a rooted tree from seed fetches; the `provenance` lint rule catches disconnected components. **Semantic search, if you want it.** `hyperresearch embed sync` populates embeddings (provider-pluggable: `voyage`, `openai`, or the default `none`, which needs zero API keys) and `search --semantic` blends vector similarity with full-text ranking. ### Curation: notes have a lifecycle Every session ends with a curation pass, and notes move through `draft` → `review` → `evergreen`, or `stale` → `deprecated` → `archive` as material ages out. That's what keeps a vault from turning into a landfill of half-read pages. ```bash hyperresearch note update --summary "..." --add-tag -j # promote a draft hyperresearch dedup -j # near-duplicate pairs by content similarity hyperresearch topic tree -j # the topic hierarchy hyperresearch index build -j # regenerate index pages hyperresearch batch set-status stale --tag -j # bulk lifecycle moves hyperresearch link --note --dry-run -j # wiki-links the linker would add ``` ### You are not locked in The vault is markdown in a directory. Everything below is a convenience on top of that, not a dependency. ```bash hyperresearch export json -o out.json # every note as structured JSON hyperresearch export vault # a filtered subset to another directory hyperresearch import # pull an existing markdown collection in hyperresearch git changed -j # notes with uncommitted changes hyperresearch git log -j # notes touched by recent commits hyperresearch watch # auto-sync while you edit in your own editor ``` --- ## Use the vault outside Claude Code **An MCP server.** `pip install hyperresearch[mcp]`, then `hyperresearch mcp` speaks stdio, so Claude Desktop, Cursor, or anything else that speaks MCP can work the same vault. Thirteen tools: `search_notes`, `read_note`, `read_many`, `list_notes`, `get_backlinks`, `get_hubs`, `vault_status`, `lint_vault`, `check_source`, `list_sources`, `fetch_url`, `create_note`, `update_note`. **A local web UI.** `hyperresearch serve --open` starts a stdlib HTTP server on port 8080 with note browsing, tag pages, search, and an interactive link graph. No build step and no JavaScript dependencies. --- ## Source ranking: quality is persistent, not vibes Every source accumulates a composite `quality_score` built from source-type tier, fetch-time utility, citation authority (from OpenAlex / Semantic Scholar, including **retraction flags**), and vault PageRank centrality: ```bash hyperresearch sources score -j # Enrich DOI-bearing notes: citations, venue, retractions hyperresearch graph rank -j # PageRank over the link + provenance graph hyperresearch search "q" --ranked -j # Quality-weighted full-text search hyperresearch sources independence -j # Cluster syndicated/derivative copies: 5 copies of one press release = 1 vote hyperresearch claims search "q" -j # Query extracted claims across all sources ``` Retracted sources are floored to near-zero quality, and a ship-time retraction sweep re-checks every cited DOI fresh, so a retraction published yesterday is caught today. Even on vault sources reused from old runs. --- ## Runs: resumable, budgeted, verified Every run owns an isolated workspace (`research/runs//`) and a manifest. Concurrent runs never collide, and a crashed run resumes exactly where it stopped: ```bash hyperresearch run status -j # Step-by-step status, spend, escalation queue depth hyperresearch run resume -j # Exact next step + Skill invocation to continue hyperresearch run report -j # Per-step wall-time / spend / source-yield telemetry hyperresearch run verify -j # Ship gate: headings, length, citation density, cite-check resolution ``` `run init --budget 50` caps estimated API-equivalent spend; crossing the cap blocks the run rather than letting it quietly balloon. And before any report ships, the verification battery runs: **quote-integrity** (every quoted span must exist verbatim in a vault note), **retracted-citations** (citing a retracted source unacknowledged blocks the ship), **numeric-consistency** (numbers untraceable to evidence get flagged), plus the cite-check step's per-citation binding audit. --- ## What's structurally enforced - **Verbatim prompt as gospel.** `scaffold-prompt` lint blocks if the scaffold doesn't open with the user's exact prompt - **Locus coverage.** Every step 4 locus must have a step 5 interim note; missing interims flag as errors - **Patch-only modification.** Steps 14, 15, 16 are tool-locked to `[Read, Edit]`. They cannot regenerate the draft - **Critical findings never silently skip.** `patch-surgery` lint surfaces any critical finding the patcher couldn't apply - **Quoted text must exist.** `quote-integrity` lint blocks any quoted span that doesn't appear verbatim in a vault note; hallucinated quotes cannot ship - **Retractions block the ship.** Citing a retracted source without acknowledging the retraction is a hard error at the final gate - **Schema integrity.** `tier`, `content_type`, and `type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index - **Hygiene leaks caught on the way out.** Scaffold sections, YAML frontmatter, and prompt echoes are stripped by step 15 before ship - **Fetched text is data, never instructions.** Web-fetched bodies are served inside an `` fence on both `note show` and `search`, so a page telling the agent to ignore its instructions is read as content --- ## The web is hostile input A research agent reads hundreds of pages it did not choose, and any one of them can contain text addressed to the agent rather than to you. Every body fetched from the web is served wrapped in `` delimiters with an inline treat-as-data preamble, on both paths that serve bodies (`note show` in single, batch, and JSON forms, and `search` with bodies included). Notes your own pipeline subagents wrote pass through unwrapped. Forged fence tags inside a fetched body are neutralized and left visible for forensics, the `url` attribute is HTML-escaped with control characters stripped, and in `search` the wrapping happens after token-budget truncation so the closing fence can never be severed. The fetcher, depth-investigator, draft-orchestrator, and source-analyst prompts all carry a policy block telling them not to launder a fenced page's directives into trusted output. Resolved URLs from third-party APIs get the same treatment. An open-access location arrives inside someone else's JSON, so it's checked for scheme, embedded credentials, and publicly-routable resolution before anything fetches it. --- ## Web providers The `[web] provider` setting in `.hyperresearch/config.toml` picks how pages are fetched and, for the providers that support it, how the web is searched. All are optional extras except the default. - **`builtin`** (default) — plain HTTP fetch with no search. Zero extra dependencies; the starting point everything else improves on. - **`crawl4ai`** — headless browser fetch with stealth, PDF extraction and the browser-escalation lane. The one the pipeline is tuned for. `pip install "hyperresearch[crawl4ai]"`. - **`exa`** — neural web search and page extraction. Needs an API key. `pip install "hyperresearch[exa]"`. - **`tavily`** — search and extraction built for agents. Needs an API key. `pip install "hyperresearch[tavily]"`. - **`parallel`** — [Parallel](https://parallel.ai/)'s Search MCP endpoint, which needs no account or key. Search only — bulk fetch waves degrade to per-URL, so it is a good search provider rather than a replacement for the crawl4ai fetch path. Every request from one process carries a random session ID that Parallel uses for correlation and rate limiting on its side. `pip install "hyperresearch[parallel]"`. ```toml # .hyperresearch/config.toml [web] provider = "crawl4ai" ``` --- ## Authenticated crawling + the browser lane Fetch from LinkedIn, Twitter, paywalled sites or anything you can log into: ```bash hyperresearch setup # Browser opens. Log into your sites. Done. ``` LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible browser to avoid session kills. **Blocked fetches escalate instead of dying.** When headless crawling hits a login wall or bot wall mid-run, the URL queues as an escalation (`hyperresearch escalation list -j`). If you have the [Claude-in-Chrome](https://claude.com/chrome) extension, the browser-fetcher agent drains the queue by driving your real, logged-in Chrome. Hard boundary: **CAPTCHAs, 2FA, and logins are never solved automatically.** They're consolidated into one message and handed to you. --- ## Scholarly discovery: eight sources, one query, one deduplicated list For any topic with a research literature, search the scholarly sources BEFORE web search. They return citation-ranked canonical works; web search returns derivative commentary. That advice used to be delivered as a list of URL templates the agent was trusted to assemble by hand — no retry, no rate limiting, no dedup, no tests. It is now a real client layer: ```bash hpr scholar search "Byzantine iconoclasm" -j # every available source, merged hpr scholar search "GLP-1 cardiovascular outcomes" --scope papers -j hpr scholar search "credit default swaps" -s edgar -s fred -j hpr scholar sources # what is wired, what each covers ``` One call queries every configured source, merges records that are the same work (by DOI first, then by normalized title and year), and returns one list. A work found by two providers carries both in `also_in`, with the higher citation count and the longer abstract. Every provider shares one cache, one per-host courtesy rate limiter, and one result shape. **The literature sources:** - **[OpenAlex](https://openalex.org/)** — ~250M works across every discipline, including books, book chapters and theses. The right default outside STEM, where the incumbent tools are weakest. - **[Crossref](https://www.crossref.org/)** — the DOI registry itself. Authoritative metadata for ~160M registered works, including registrations too new for anything else to have indexed. - **[CORE](https://core.ac.uk/)** — the largest open-access full-text aggregator. Hosts the text rather than linking to it, so it is also a full-text resolver (below)