Add documentation guides for LLM-based web scraping frameworks

[vdusek]

Summary

Our current documentation guides cover traditional web scraping libraries (BeautifulSoup, HTTPX, Playwright, Selenium, Scrapy, etc.), but we're missing guides for modern LLM-based web scraping frameworks that are becoming increasingly popular.

We should add new guides showing how to use these frameworks with the Apify SDK.

Guides to add

• Crawl4AI – LLM-friendly web crawler and scraper with built-in support for structured extraction
• Scrapling – high-performance, adaptive web scraping library with intelligent content extraction
• Browser Use – AI agent framework for browser automation using LLMs
• ScrapeGraphAI – LLM-powered scraping pipelines using graph-based logic

Notes

• Each guide should follow the same structure and conventions as the existing guides (e.g. BeautifulSoup, Scrapy)
• Include a practical example demonstrating integration with the Apify SDK (Actor input, storage, etc.)
• Related issue for actor templates: Add Python actor templates for LLM-based web scraping frameworks actor-templates#739

Edit

• Removing ScrapeGraphAI as it is probably not relevant according to the PyPI download stats.

[m13v]

This is a solid roadmap. Browser Use especially deserves a thorough guide since it's becoming the default choice for LLM-driven browser tasks.

One thing worth covering in the Browser Use guide: how to handle concurrent browser sessions safely. When you scale up browser agents on Apify (multiple actors running in parallel), you hit session isolation issues fast. Cookies leak between contexts, login states get corrupted, and rate limits stack up across instances sharing the same IP.

We dealt with this extensively while building Fazm (a macOS AI agent that automates browser tasks). The pattern that scaled was: one persistent browser profile per task, with a file-based lock to prevent two agents from touching the same profile simultaneously. Without that lock, Playwright's context isolation alone isn't sufficient for production workloads.

Also worth noting for the ScrapeGraphAI guide: their graph-based pipeline design pairs well with Apify's storage primitives. The intermediate extraction nodes can write directly to Apify datasets, which makes debugging much easier than piping everything through the LLM in one shot.

[m13v]

Here's the browser session locking approach I mentioned for preventing profile corruption with concurrent agents: https://github.com/m13v/browser-lock/blob/main/playwright-lock.sh

This is how we handle tool execution in our AI agent (including browser automation calls): https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift