Meta 的 Muse 在网页抓取上表现出色？HN 用户讨论其能力与封禁困境

Domain: sigh.dev Points: 36 | Comments: 39 [Top Comments] - I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles). Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing. - By describing it here you've already released it no? - FYI Muse smashes through those captchas natively without prompting. - Right, but what happens when everyone uses that at scale? Without agreements and standards it isn't pretty. - their static ip's were initially good and didn't get flagged, but now most sites are recognizing their ip ranges and blocking. Muse's utility has significantly dropped with the blockages. To become truly useful again they will need to use residential proxies, but I can't see them use those due to the risks and reputational damage. - they can just use the user ip. i think grok already does this. - Most residential proxies are already *far* more blocked and rate limited than any Meta IP. The internet is becoming a very weird place, where individual and "trusted" personal IPs are becoming a kind of commodity. Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised. AI analysis is turning this up to 11. - Muse ran into a captcha and asked me if I wanted it to solve it. So of course I clicked yes and it dutifully convinced the site that it was not a bot. - For 2(3?) decades we've been training the robots to tell traffic lights from fire hydrants. It's finally paying off. - I think this is the end of sites where it is expected that only humans may interact with them. It's been cat and mouse for a while, and some places like Reddit sell API access, but these agents are for all intents and purposes, humans interacting with the site. They're going to need to figure out new business models. - As someone having to fight Meta's bots everyday to keep websites accessible to actual customers, I'm not surprised to read that it can be seen as something positive on the other side of the fence. But I'm wondering: at what cost? - I think for my own side projects i would require the user to login if they were making requests from those ip addresses or block. - Im guessing more of the internet will be login walled from now on.