最新资讯
bloomberg.com Nvidia 加强东南亚芯片追踪以打击走私 thestack.technology 模型路由需求升温,Agent Router 推出开源替代方案 unite.ai Cashfree 为 AI Agent 的托管结账加入 WebMCP 工具 dev.to 我们在 DolphinBench 上测试记忆检索:纯关键词搜索暴露了自身回归 dev.to 别再把 Agent 写成 while 循环,把它建模为状态机 analyticsindiamag.com 小语言模型可能重塑企业 AI 部署方式 bloomberg.com Rightmove 与 Scout24 借 AI 产品应对低迷房市并上调展望 stackoverflow.blog 欢迎来到芯片时代:半导体制造、AI 数据中心与芯片设计的未来 www.youtube.com AI Agent 总犯同样的错:CentralAgent AI 将记忆重构为经验系统 pub.towardsai.net Netflix 科学家建议重采样 LLM 评判:5 次仅将误差棒缩小 13% pub.towardsai.net Jev 作为 LLM 前的静默层:构建提示感知的模型路由器 reddit.com OpenAI 称 Astra 仅满足 10% 安全任务标准
多元动态
登录 注册
行业动态 dev.to · @Dexterlung

Google 抓取了 270 篇文章却零收录:它收到的正文是 0 个字符

原文 我的收藏

作者从 Google Search Console 导出 427 行数据,发现 270 条为 crawled–currently not indexed、157 条 Soft 404、95 条因 canonical 选择不同而未收录。他提出不要先猜关键词或外链,而应以 Googlebot 身份抓取自己的页面,检查实际返回内容。文中称 Google 收到的正文为 0 字符,但片段未给出根因与修复后的验证结果。

AI推荐理由 作者用 Search Console 导出数据(427 行,270 条 crawled-not-indexed、157 条 Soft 404、95 条 canonical 冲突)说明问题,并给出以 Googlebot 身份抓取自身页面的排查方法,属于可复现的实操经验;但正文片段在关键处截断,未展示 0 字符的根因与修复结果,结论仍待验证。

62

正文

Read on: the acceptance check that proved 1 = 1 · 繁體中文版 I assumed my content wasn't good enough. Google had never read my content. What it received inside was zero characters. The numbers that started it On 13 August 2026 I opened Google Search Console and exported two reports, 427 rows. Three lines stood out: Status Rows Crawled – currently not indexed 270 Soft 404 157 Duplicate, Google chose a different canonical 95 "Crawled – currently not indexed" has an unambiguous meaning: Google came, looked, and declined. That is far worse than not being crawled. Not crawled means it hasn't arrived yet. Crawled and declined means it read the page and judged there was nothing worth indexing. My first instincts were keywords, depth, backlinks. All three were the wrong direction, and all three would have cost me another three months of writing articles nobody could find. There was only one correct first move: stop guessing what it sees and go be it. First move: fetch your own page as Googlebot I requested my own article URL with Googlebot's User-Agent, saved the raw HTML, and counted two things: characters inside → 0 JSON-LD blocks → 0 Repeated it with GPTBot. Identical. So besides Google, the AI-search side was blind too. The cause is unsurprising to anyone who has shipped a single-page app: the first HTML the server sends is a shell, and the body appears only after JavaScript runs. Google says it executes JavaScript, which is true — but "will execute" and "will always wait for you to finish before scoring" are different promises. The part I want to emphasise isn't the conclusion. It's how long it took me to run that command. That HTML was one curl away the entire time. I wrote 270 articles before I fetched it once. Fix 1: put the body in the first-hand HTML My site already had a Netlify edge function doing og-tag injection, which means it already reads the article from the database for social previews. So body injection needed no new architecture — three more things on the same trip: render the article body to semantic HTML and place it inside emit JSON-LD (BlogPosting) emit hreflang, and only when the other language version is actually published The human path is untouched; the branch is bot-only, so real readers pay nothing. One technical decision deserves its own paragraph: I did not reuse the frontend's markdown renderer. It sanitises with DOMPurify, DOMPurify needs a DOM, and the Deno edge runtime has none. So I wrote an escape-first renderer on the edge side: escape every character, then apply structure. That ordering makes XSS structurally impossible rather than "unlikely because I was careful." The cost lands on raw-HTML blocks, which degrade to plain text. I didn't estimate whether that mattered — I asked the database. Of 1,112 text blocks in published articles, 1,111 use markdown and 1 uses raw HTML. I'm pointing at that because it is the most useful discipline I picked up this year: when you are about to make a judgement about "all of the X", derive the scope from a query, not from memory. Had I written "most of them are probably markdown" I would likely have been right — and I would not have known the 1 existed. (The same habit, learned the hard way, is what fixed a privilege sweep that reported zero residue while missing seven views.) Verification: 28 unit tests, 12 cross-layer contract tests, all green. Green isn't the evidence, though — I also built 9 deliberately broken variants and confirmed each turned exactly the check it should red, and that restoring made everything green again. 9/9. One self-inflicted trap worth half an hour of your life: I used a sentinel character to mark inline-code boundaries, and the first version used a literal NUL byte. Git immediately classified the file as binary (Bin 0 -> 21694 bytes) — no diff, no line-by-line review. An escape sequence carries identical semantics and stays readable. Fix 2: 157 soft 404s, because the site answered 200 to everything The second number is worse, because it grows by itself. I picked two URLs out of the GSC export and requested them by hand: /product/panama-esmerlada-geisha-washed/ → 200 + homepage HTML /product-tag/washed-processed/ → 200 + homepage HTML Enter fullscreen mode Exit fullscreen mode Both are dead WordPress-era URLs. But the last line of my _redirects was: /* /index.html 200 Enter fullscreen mode Exit fullscreen mode Every SPA deployment guide tells you to add that line — without it, refreshing any sub-path 404s. Its side effect: every dead URL is also caught by it, and answered 200. Google receives a 200 and something that looks like a homepage, and files it as a soft 404. Every mistyped or mis-pasted link manufactures another one. Two layers of fix: edge function: for /content/:slug, /product/:id, /works/:slug, when the record definitively does not exist, return status 404. The body is still the SPA, so the visitor still sees the in-site "page not found" screen; only the status code becomes honest. _redirects: legacy WooCommerce/WordPress archive paths return 410 Gone. Not a 301 to /shop — in Google's eyes that is still a soft 404, just reported somewhere else. Seven legacy .md URLs get a 301 stripping the extension. The load-bearing part isn't the 404, it's fail-open This is the easiest thing in the batch to get wrong, and getting it wrong produces no visible symptom. My data helper fetchRows returned an empty array for both "query failed" and "zero rows found". While it only did injection, that was harmless — both cases meant "don't inject." The moment "no record → return 404" existed, the same code meant something else entirely: database hiccups for 3 seconds → my real article is declared nonexistent → that 404 lands in Google's index. The fix is to return { ok, rows } and treat only ok-and-zero as genuinely missing. /product/:id needed one more layer: decide locally whether the string even looks like a uuid. A non-uuid sent to PostgREST returns 400, and 400 is "query failed", which fail-open lets through. The 82 legacy WooCommerce permalinks I needed to block are all non-uuid. They must be judged locally, not shipped out as a question. Four mutations for this piece: remove fail-open, remove the uuid check, drop a header cleanup in the 404 branch, revert fetchRows to a bare array. The third one did not go red the first time. My assertion sliced the file contents to the end, so it matched a headers.delete further down in the injection path rather than the one in the 404 branch. Classic "the substring had another source" — the test appears to check one thing and is being fed by somewhere else. Slicing to just the 404 branch made it 4/4. I did not think that defect up. The mutation exposed it. Fix 3: one stylesheet costing 2,222 ms Indexing handled, on to speed. Local Lighthouse against one of my article pages: Render-blocking resource Delay Size fonts.googleapis.com/css2?Noto+Sans+TC 2,222 ms 98 KiB assets/index-*.css 341 ms 26 KiB assets/vendor-formkit-*.css — 6 KiB One Google Fonts stylesheet, 2,222 ms on its own. Of those 98 KiB, 92 KiB are unicode-range subset declarations I never use (CJK fonts split into dozens of subsets, one declaration each). I thought I had handled this, because the HTML contains . But the load-bearing line is the rel="stylesheet" right beneath it — preload only makes the browser fetch earlier; it does not take the stylesheet off the critical rendering path. With both present, the slow one is still slow. The fix swaps rel to stylesheet on load. No flash of invisible text, because the font URL already carries display=swap and the splash screen uses system fonts. JavaScript-disabled clients get the ordinary tag inside . And afterwards I went back and confirmed all three injection anchors from Fix 1 were still intact. Touching fonts shouldn't affect body injection — but "shouldn't" and "confirmed" are different words. Fix 4: a 531K PDF bundle preloaded on every page The live index.html had a modulepreload line pointing at vendor-pdf-*.js, 531K. The three pages that use PDF (poster studio, sticker generator, cupping station) are all lazy routes. Their dependencies have no business in the first screen. Reading the build output made it clear: every page chunk in dist — including "page not found" and the newsletter page — carried a bare import"./vendor-pdf-*.js", and that chunk exports three identifiers nobody binds. So this wasn't "somebody needs pdf." It was my manual chunking rule, matching on strings, fighting Rollup's own splitting. Fix: return undefined for those three packages and hand the decision back to Rollup. ⚠️ The first version just deleted the line, and made things worse: the return 'vendor' fallback below caught them and moved them into the core vendor bundle, which loads earlier. Core vendor went 499K → 1,057K and precache 2,398 → 2,930 KiB. You have to return undefined explicitly, before the fallback. That was caught by verifying the build output, not by thinking. ⚠️ The first verification script was also wrong. It asserted "the pdf chunk must have no static importer", which flagged two legitimate cases — a lazy route's chunk statically importing a shared chunk is normal, because that whole subtree only loads on navigation. The real criterion is whether the transitive static-import closure from the entry point reaches it. Computing the closure fixed it. A guard that barks at legitimate change is as bad as one that never barks. Final numbers: Before After modulepreload 6 (incl. 531K vendor-pdf) 5 core vendor 499K 419K precache 2,398 KiB 2,319 KiB first-screen static closure — 6 chunks / 1,461K, pdf not among them Then I shipped a new defect: the title, three times With four layers done, I thought the batch was finished. The next day I looked at a real screenshot of the live article page. It opened with: title → date → summary → title (as a figure caption) → title (h1) → summary Title three times, summary twice. Shipped by my own previous commit. The cause is simple and unflattering: I designed a layout without reading the frontend component first. Its contains exactly one line, a — the title, hero image and summary are already inside the blocks. My injector then added an h1, a date, a summary and a hero image with a caption. All four duplicates. The fix is to render the blocks and nothing else — faithfully mirroring the existing renderer. The test was an accomplice I had a test asserting "the output contains exactly one h1". It was green throughout, while production grew three. The reason was the fixture: its blocks happened to contain no title block. So the assertion held forever, over data that didn't resemble reality. When the fixture doesn't look like the real thing, the test will vouch for the defect. I gave the fixture a hero block (the real shape) and added a regression test specifically for duplication. Incidentally, my mutation tool caught its own stale config in that round — the string it was supposed to mutate no longer existed, so it reported "this proof is invalid" instead of counting it as a pass. That is what a guard should do. Final acceptance: 34 unit tests, prove-red 11/11. Two sentences carry this section: An injection layer's job is to mirror the existing renderer faithfully, not to redesign the layout. "Measuring" is not "looking." No Lighthouse score will ever show you a title printed three times. A screenshot showed it in three seconds. A footnote: a byline can't be a shell In the JSON-LD I changed the author type from Organization to Person to feed E-E-A-T person signals. But I left the brand name in the name field. Google now saw "a person named after a brand." Right type, wrong load-bearing field. I changed it to my real name and added an assertion that can actually fail: author.name !== SITE_NAME. Before that, the test only checked @type === 'Person' — which stays green if you put the brand name back, and that is exactly the defect this change existed to fix. If you run an SPA content site, check in this order Twenty minutes, roughly: Fetch one of your own article pages with Googlebot's User-Agent and count characters inside . Cheapest cut, and the origin of everything else. Export the soft 404 list from GSC and request two of them by hand. If your _redirects has /* → /index.html 200, you already know the answer. Run Lighthouse and read only the render-blocking section. On a CJK site the top entry is very likely a font. Open the live index.html and read the modulepreload lines. Ask each one whether the first screen truly needs it. When you're done, look at a real screenshot. Not the score. The picture. The last thing I'll leave here: what actually saved me in this batch was not any individual fix — it was prove-red, deliberately breaking the code and watching whether the check turned red. It caught three false green lights in this batch alone (an assertion matching a string from elsewhere, a guard flagging legitimate cases, a fixture that didn't look real), and not one of those was avoidable by being more careful. Where the numbers come from GSC exports, 2026-08-13, 427 rows: crawled–not-indexed 270 / soft 404 157 / duplicate 95 Live fetches as Googlebot and GPTBot: 0 characters, JSON-LD 0 Local Lighthouse 13, target /content/the-mechanism-i-never-checked Build output: modulepreload 6→5, core vendor 499K→419K, precache 2,398→2,319 KiB Prove-red totals: body injection 9/9, 404 honesty 4/4, layout duplication 11/11 Originally published on my blog: Google crawled 270 of my articles and indexed none — the body it received was 0 characters I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit

来源报道 · 1 篇 · 以下链接均前往原始媒体

← 返回精选动态 原文

更多内容