Scrapling v0.5.1 发布：增强 AI agent 浏览器控制与远程会话

Official release notes (https://github.com/D4Vinci/Scrapling/releases/tag/v0.5.1) **Small version bump, big browser upgrade! This version lets AI agents follow motion and control drag and drop, adds dedicated remote browser sessions, and brings fixes across parsing, spiders, proxies, and the shell 🚀** > [!NOTE] > **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)** > [!WARNING] > **This release introduces breaking changes to remote browser connections. Connecting to remote browsers now requires a new browser session class. Check the [remote-session migration guide](https://scrapling.readthedocs.io/en/latest/migrating-to-v0.5.html#move-cdp-connections-to-remote-sessions) before updating.** ## 🚀 New Stuff and quality of life changes - **AI agents can now follow motion and take finer control of the mouse** (Check the [docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)): - **Watch the page in motion**: a short sequence of images helps the agent understand animations, moving controls, and changes that a single screenshot can miss. - **Drag, inspect, adjust, and drop**: hold a mouse button, move an item or slider, inspect the page while still holding it, then adjust and release. Supports left, right, and middle buttons. - **Reuse a logged-in remote browser** or start with a separate browser context, so the agent can choose the right session for the job. - **Spend fewer tokens on tool descriptions** with shorter text and less repeated guidance. - **Dedicated remote browser sessions with control over login reuse**. `RemoteSession` and `AsyncRemoteSession` connect to an existing Chromium browser through an HTTP or WebSocket CDP endpoint. They support the usual stealth session features, including Cloudflare solving. Both classes are also available in the interactive shell. (Check the [docs](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html#3-cdp-connection)) ```python from scrapling.fetchers import RemoteSession with RemoteSession(cdp_url="http://localhost:9222", new_context=False) as session: page = session.fetch("https://example.com") ``` - `new_context=False` reuses the browser's existing default context and login state. Its context settings stay in use, and its existing tabs remain open when Scrapling disconnects. - The default, `new_context=True`, creates a separate context and closes it when the session ends. Scrapling uses its own tabs in either mode. - **Choose your spider's log file encoding** with `log_file_encoding="utf-8"` or another encoding. The default, `None`, keeps Python's default encoding. (Check the [docs](https://scrapling.readthedocs.io/en/latest/spiders/advanced.html#logging)) ## 🐛 Bug Fixes - **XPath expressions now return their values correctly**, including `string()`, `count()`, and `boolean()`. Empty strings, zero, and false remain available through the usual selector API, by @jayhemnani9910 in [#552](https://github.com/D4Vinci/Scrapling/pull/552). - **Fixed parsing of HTML with no root element**, including whitespace-only, comment-only, and doctype-only input, by @rushilnimbalker250 in [#536](https://github.com/D4Vinci/Scrapling/pull/536). - **Fixed `find_similar(similarity_threshold=0)` when the reference has no attributes to compare**, by @Master-Norna in [#539](https://github.com/D4Vinci/Scrapling/pull/539). - **Fixed duplicate robots.txt fetches when concurrent requests reach the same domain**, by @RayenMlayeh in [#538](https://github.com/D4Vinci/Scrapling/pull/538). - **Fixed temporary-file collisions during concurrent development-cache writes**, by @xixihaha-ha in [#546](https://github.com/D4Vinci/Scrapling/pull/546). - **Malformed links no longer stop extraction of valid links on the same page**, by @dajiaohuang in [#548](https://github.com/D4Vinci/Scrapling/pull/548). - **Fixed CSV feeds that use carriage returns as row separators**, while preserving line breaks inside quoted fields, by @xunguangzlj-cloud in [#551](https://github.com/D4Vinci/Scrapling/pull/551). - **Truncated or damaged gzip feeds now follow the normal warning-and-skip path** in sitemap, XML-feed, and CSV-feed spiders, by @cangen185-sudo in [#522](https://github.com/D4Vinci/Scrapling/pull/522). - **Fixed browser proxy authentication with percent-encoded usernames and passwords**, by @kevin9327 in [#545](https://github.com/D4Vinci/Scrapling/pull/545). - **Improved browser cleanup when session setup or context closing fails**, releasing owned resources while keeping borrowed remote contexts open. - **Stopped shell and HTTP CLI cookie parsing from dropping browser cookie values with spaces or quotes**, including the cookies that follow them, by @waw666waw666 in [#541](https://github.com/D4Vinci/Scrapling/pull/541). Fixes [#540](https://github.com/D4Vinci/Scrapling/issues/540). - **Curl conversion now accepts `-L` and `--location`**, keeping the existing redirect behavior, by @jayhemnani9910 in [#554](https://github.com/D4Vinci/Scrapling/pull/554). - **Fixed `scrapling extract fetch --ai-targeted` not enabling ad blocking as documented**, by @postoso in [#527](https://github.com/D4Vinci/Scrapling/pull/527). _🙏 Special thanks to the community for all the continuous testing and feedback_ --- ### Big shoutout to our Platinum Sponsors Repository README (current repository documentation, not release changes) Effortless Web Scraping for the Modern Web Selection methods · Fetchers · Spiders · Proxy Rotation · CLI · MCP Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library. Scrape static pages, render JavaScript, interact with websites, and extract data from the API calls behind them. Build crawlers with reusable sessions, proxy rotation, automatic retries, pause/resume, and built-in data exports. Use it from Python, the terminal, or your AI agent through its MCP server and Agent Skill. From extracting a few fields to collecting a whole website as clean Markdown, Scrapling covers the full scraping workflow. ## What you can do with Scrapling - **[Fetch & interact](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture. - **[Parse & extract](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output. - **[Crawl & export](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports. - **[Scrape with AI or from your terminal](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell. Start with a single page: ```python from scrapling.fetchers import Fetcher page = Fetcher.get('https://quotes.toscrape.com/') quotes = page.css('.quote .text::text').getall() print(quotes) ``` Or scale up with a spider: ```python from scrapling.spiders import Spider, Response class QuotesSpider(Spider): name = "quotes" start_urls = ["https://quotes.toscrape.com/"] async def parse(self, response: Response): for quote in response.css('.quote'): yield { "text": quote.css('.text::text').get(), "author": quote.css('.author::text').get(), } result = QuotesSpider().start() print(len(result.items)) result.items.to_json("quotes.json") ``` This prints `10` and saves the quotes and authors to `quotes.json`. # Platinum Sponsors NodeMaven - The most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. Use code SCRAPLING35 for 35% discount. Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, Hyper Solutions provides API endpoints that generate valid antibot tokens for Akamai, DataDome, Kasada, and Incapsula. Simple API calls, no browser automation required. Evomi : residential proxies from $0.49/GB. Scraping browser with fully spoofed Chromium, residential IPs, auto CAPTCHA solving, and anti-bot bypass. Scraper API for hassle-free results. MCP and N8N integrations are available. TikHub.io provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets. Also offers DISCOUNTED AI models - Claude, GPT, GEMINI & more up to 71% off. Read a full review of Scrapling on The Web Scraping Club (Nov 2025), the #1 newsletter dedicated to Web Scraping. Swiftproxy provides scalable residential proxies with 80M+ IPs across 195+ countries, delivering fast, reliable connections, automatic rotation, and strong anti-block performance. Free trial available. Byteful supplies ethical residential, mobile, ISP, and datacenter proxies, plus scraping and AI automation tools. Use code SCRAPLING10: 10% off residential proxies. ProxyLane: 90% fewer CAPTCHAs on your Scrapling runs. 28M real P2P residential devices, 98.7% clean IPs, 195 countries. City/ISP/ZIP targeting, sticky/rotating sessions, traffic never expires. Try 350 MB for $1.95. Code SCRAPEANDTAKE: 25% off. LundProxy: Stop paying per GB. Unlimited residential traffic, 100M+ IPs, 190+ countries. Pay by bandwidth. Request a free 50 Mbps residential proxies trial. Resolvr: Scraping at scale means CAPTCHAs at scale. Resolvr solves reCAPTCHA and Cloudflare challenges on real devices through one API. Free beta access is available now. hCaptcha in your Scrapling runs? NoneCap returns valid hCaptcha tokens, Enterprise sitekeys included, through one API. 1,300 free credits on signup. $1/GB at DataImpulse. Ethical first-party proxies. 90M+ IPs, 195+ countries. Residential, datacenter, mobile. Pay-as-you-go = no subscription. Traffic never expires. Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646) # Sponsors Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci) and choose the tier that suits you! --- ## Key Features ### Spiders - A Full Crawling Framework - 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects. - ⚡ **Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays. - 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID. - 💾 **Pause & Resume**: Save pending requests and seen URLs with `crawldir`. Press Ctrl+C to stop gracefully and restart with the same directory. - 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls. - 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic. - 🚦 **AutoThrottle**: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what `Retry-After` asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops. - 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching. - 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers. - 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, `XMLFeedSpider`/`CSVFeedSpider` for iterating XML/RSS and CSV feeds, and `ShopifySpider` to collect products from public Shopify stores, one item per variant. - 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own. - 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`. ### Advanced Websites Fetching with Session Support - **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3. - **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `StealthyFetcher` class supporting Playwright's Chromium and Google's Chrome. - **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation. - **Session Management**: Persistent session support with `FetcherSession` and `StealthySession` classes for cookie and state management across requests. - **Proxy Rotation**: Built-in `ProxyRotator` with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides. - **Domain & Ad Blocking**: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers. - **DNS Leak Prevention**: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies. - **Remote Browsers**: Instead of launching a browser locally, connect to one that's already running through CDP with `cdp_url`, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build with `executable_path`. - **Background API Capture**: Enable `record_requests=True` on a browser session, then search its saved responses with `session.network.search(limit=None)`. Read API data and HTTP errors across page loads without sending requests again. History is bounded and saves text response bodies. - **Async Support**: Complete async support across all fetchers and dedicated async session classes. ### Parsing and Data Extraction - 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms. - 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more. - 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements. ### AI Features - 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with HTTP requests, browser actions, screenshots, saved login state, and background API responses. Uses our stealthy browser with the right tools to enable suitable AI models to bypass any CAPTCHA is faces while working (not to mention our Cloudflare solver). It uses a lot of methods to reduce the content sent to the model, block ads, and protect it against Prompt-Injection. Remote browsers over CDP and HTTP transport with authentication options are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE)) - 🧠 **Agent Skill**: A ready-to-install [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing. - 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `ContentSpider` template, all without an LLM in the loop. ([docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html)) ### High-Performance & battle-tes