从服务器端读取小红书笔记究竟需要什么

Originally published at linkdigest.dev, where I build this. The problem, in one paragraph Paste a Xiaohongshu, Douyin, TikTok, YouTube or X link into an AI agent and it fetches the URL, gets an app-download shell or a login wall, and tells you there is nothing there. It is not wrong. The content of those posts is video, images, and text printed inside images, behind tokenised share links — none of it is in the HTML a fetch returns. Strip the tags from a real Xiaohongshu note and 264 characters of navigation are left. What LinkDigest does about it LinkDigest is a hosted reader: one call turns the link into text an LLM can use — a transcript with timecodes, the on-screen text, a description and OCR of every image, the caption and metadata — as Markdown or JSON. It is an MCP server (claude mcp add --transport http linkdigest https://linkdigest.dev/mcp --header "Authorization: Bearer $KEY"), a REST API, and a web console. Measured on a 17-image Xiaohongshu note: 17 images described and read, 381 on-screen text fragments, 13 key points, 119 seconds. Three digests are free, no card. What it does not do, stated up front: Bilibili refuses our server's address (HTTP 412), Instagram is wired but not verified, Facebook is out of scope. A digest that could not read part of a post says so in a degraded field instead of returning a thin result quietly — and a digest that read nothing costs nothing. Every claim here was checked against a live link. Where something doesn't work, it says so. Paste a Xiaohongshu link into an AI coding agent and it sees nothing. Same for Douyin. The usual answer — "just use yt-dlp" — is half true in a way that wastes an afternoon, so here is the whole picture. yt-dlp only reads half of Xiaohongshu XiaoHongShuIE reads note.video.media.stream. That is a video note. Image notes — 图文, a caption plus a stack of photos — are the majority of the platform and usually the ones worth reading, and the extractor has nothing to say about them. Getting those means parsing the note payload out of the page itself: the state blob the page ships, and the imageList inside it. Reusing yt-dlp's own js_to_json and traverse_obj for that keeps you aligned with upstream when the page shape shifts, which it does. The page is different depending on who you say you are This one cost me an embarrassing amount of time. Request a note with a mobile user agent and you get 202KB of app-download shell whose is just the site name. Request the same URL with a desktop user agent and you get 85KB containing the real note. Both are HTTP 200. Nothing tells you that you got the wrong one except that the content isn't there. [Diagram: The same note URL returns 202KB of app shell to a mobile user agent, and 85KB containing the note to a desktop one.] Cold requests are rejected Fetching a note URL directly, with no prior session, does not work. Fetch the /explore feed first, keep the cookie jar it gives you — acw_tc, abRequestId — and then the note loads. No account, no credentials, no API key. Just the same two-step a browser performs without you noticing. Douyin does not negotiate Coverage and output for Douyin: platforms/douyin. Douyin refuses anonymous requests outright: captcha on the web page, 403 from the APIs. There is no user-agent trick here. That path needs cookies from a logged-in session, and yt-dlp does not support the platform at all. YouTube blocks your server, not your laptop Coverage and output for YouTube: platforms/youtube. The single most common "it worked locally and broke in production" report. YouTube blocks datacenter IP ranges, so a yt-dlp fetch that is perfect on your machine fails from EC2, App Runner, Lambda or anywhere else you deploy — and no amount of configuration fixes an IP-range block. The workable fallback is a model that watches the video and returns a transcript, which costs meaningfully more than parsing captions and is worth measuring separately. Where it stops Being honest about this saves everyone time: Bilibili returns HTTP 412 to a datacenter address. Needs a residential proxy. Instagram and Facebook need a logged-in session for most posts. TikTok resolves fine but rate-limits under load; a frame fetch can come back 403 mid-job. The part nobody mentions: failure has to be loud The bug that taught me the most had nothing to do with extraction. A note was digested during a provider rate-limit storm. Every vision batch failed, the per-batch error handler swallowed each one, and the job returned images: 0, ocr: 0 — a perfectly well-formed, completely empty result. It was then written to a shared cache with a 30-day TTL. Long after the underlying problem was fixed, that link still returned nothing, because a cached answer is cheaper to serve than to recompute. The pipeline was healthy. The cache was serving a fossil, and nothing downstream could tell the difference between an empty result and an easy one. Two things came out of that: A step that loses content has to say so. The result now carries what failed, so a thin answer is distinguishable from a genuinely short post. Thin results get a short TTL, not the full one. Not "don't cache" — a link that fails every time would then re-run the full paid pipeline on every request forever. An hour bounds the staleness and the spend together. If you are building anything that caches derived content, that second one is the trap. The obvious fix is the expensive one. The findings are the same whether you use a hosted service or write your own. For how this fits into an agent's workflow end to end — the MCP tool, the job id for long media, what the output looks like — see Reading links your agent can't open.