Extract data with Crawlee-python, a library for startup founders, powered by 9.3k+ GitHub stars, for web scraping and automation.
9,272 stars765 forksPythonQuality 8/10Updated 7/1/2026100% free Β· open source
What it does
Crawlee-python is a web scraping and browser automation library for Python that allows startup founders to extract data from websites and download files in various formats.
Install / run
pip install crawlee
When to use it
β’When you need to extract data from a website that doesn't provide an API
β’When you want to automate interactions with a website, such as filling out forms or clicking buttons
β’When you need to download files from a website, such as HTML, PDF, JPG, or PNG files
Quick start
1Import the Crawlee library in your Python script: `from crawlee import Actor`
2Create a new actor: `actor = Actor()`
3Define a scraper function using the `@actor.task` decorator: `@actor.task def scrape_page(url): ...`
4Run the actor with the `actor.run` method: `actor.run()`
5Configure proxy rotation by setting the `proxy` option: `actor.settings.proxy = 'https://api.crawlee.dev/proxy '`
Ready-to-paste prompt
Extract all links from a webpage using the `fetch` function and Parsel: `from crawlee import Actor; from parselp import Selector; actor = Actor(); @actor.task def scrape_page(url); async def scrape_page(url): html = await fetch(url); selector = Selector(html); links = selector.css('a::attr(href)').get(); yield links`
Heads up: Make sure you have the correct version of Python installed, as Crawlee-python requires Python 3.7 or later, and also be aware that some features, like Playwright, require additional dependencies to be installed
Saves to your device
Topics
apify
automation
beautifulsoup
crawler
crawling
headless
headless-chrome
parsel
pip
playwright
python
scraper
scraping
selenium
web-crawler
web-crawling
web-scraping
What's inside β free to inspect
No purchase needed
Read the entire source before you build β unlike paid marketplaces that hide it behind a buy button.