How to run the 2scraper open-source scrapers: setup, proxies and the Browser API

2scraper is a GitHub organization with open-source Python scrapers, one repository per website. On 2026-10-29 there were 65 of them. Each one reads a public site without logging in, writes one row per item to JSON or CSV, and reports how the run ended. The code is MIT-licensed, and every repository documents what was measured, when, and what the scraper needs besides a local browser.

The site-specific guides in this section (SHEIN, Lidl, TipRanks, Transfermarkt, YouTube, Google Play, Avito) each cover one repository in depth. This guide covers what they have in common, so you can take any scraper from the catalog below and run it without reading another tutorial first. It explains how the repositories are laid out, how to install and configure them, how to read their output and exit codes, and when a site needs a proxy, a captcha solve or a hosted browser.

Each repository's README describes that scraper exactly. The conventions below are family defaults, and where a README differs, follow the README.

What all the scrapers have in common

One repository per site, one row per item

amazon-scraper reads amazon.com, zimmo-scraper reads zimmo.be, and so on. Each run produces a results file (JSON by default, CSV with --format csv) where every row is one product, listing, post, video or player. Fields come in the same order in both formats, and a missing value is null.

Three browser engines

Most repositories ship the same scraping loop behind three drivers: playwright_scraper.py (primary, recommended), selenium_scraper.py and puppeteer_scraper.py (pyppeteer or Node Puppeteer, depending on the repository). The drivers share the site-specific parsing in product_parser.py and, where the repository has it, the page-state logic in page_flow.py. Playwright gets the most testing. The other two are kept working, but not every path is verified end to end; the README lists which ones were run live.

A browserless path when the site allows it

When a site's own endpoint returns the data, as on Hacker News or for parts of YouTube via the Innertube API, the repository calls it directly and says so. A browser is used only when page scripts build the data.

Offline tests and a live canary

Each repository has an offline test entry point (python3 smoke_test.py or pytest tests/) that checks parsing against sanitized fixtures without network access. Most also have a scheduled GitHub Actions canary that runs the scraper against the live site and checks what comes back. Both show as badges at the top of the README. A green canary tells you the site looked the same at the last scheduled run.

Dated measurements

When a README says a site refuses datacentre addresses or needs a headful browser, it also says when and from where that was measured. Sites change, so re-check before you build on an old measurement.

Installation

The layout is the same everywhere: a virtual environment per engine (their dependencies conflict), one requirements file per engine, and a .env file for anything secret.

git clone https://github.com/2scraper/<site>-scraper.git
cd <site>-scraper
python3 -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements-playwright.txt
playwright install chromium
cp .env.example .env

For Selenium or Puppeteer, create a separate virtual environment and install requirements-selenium.txt or requirements-puppeteer.txt. Many repositories also include a Dockerfile that builds an image with Playwright and Chromium preinstalled.

The family baseline is Python 3.9 or newer; some READMEs require a later version.

Configuration: .env, environment, CLI

Keep credentials in .env or in named environment variables. Most scrapers also accept them as flags, but anything on the command line ends up in shell history and process lists. Values are taken in this order: an explicit flag, then an exported environment variable, then .env, then the built-in default. An empty value or an unfilled placeholder counts as not configured and never switches on a paid path.

Variable names follow one pattern. The exact set for a repository is in its .env.example:

Two rules apply to these values. A proxy is never swapped under a live browser session: rotation starts a fresh session without the old cookies. And a hosted CDP browser is never combined with a local proxy, user agent or fingerprint.

Running a scraper

Every scraper is a command-line tool. You tell it what to read (a search term, a category, a URL or an identifier, depending on the site) and how much:

# a search, capped at 60 results, JSON to the default file
python3 playwright_scraper.py --query "linen pants" --max-results 60

# a specific URL, saved as CSV
python3 playwright_scraper.py --url "https://example.com/some/listing" --format csv --out listing.csv

Defaults are small, typically 20–30 results and a couple of retries, so a first run is quick and, on sites with paid paths, cheap. --help lists every flag, and the README's Quick start shows the commands that matter for that site.

Some sites refuse headless Chromium, and the README says which. On a server with no display, run those under a virtual display:

xvfb-run -a python3 playwright_scraper.py --query "..."

Reading the output: the .meta.json sidecar

A run that collects data writes two files: the results (<out>.json or <out>.csv) and a sidecar, <out>.meta.json. Automation should read the sidecar, which records at least:

• status and stop_reason: whether the run completed and, if not, why (blocked, rate_limited, remote_api_error, failed_pages, parser_found_nothing and so on);
• row counts, the total the site reported where it reports one, and whether --max-results capped the run;
• paid-path usage, for example solves_spent;
• a hash of the results file, so a sidecar can be matched to its data.

A run that collects zero rows writes neither file, so the previous good output stays in place. Pass --allow-empty if you want an empty file on purpose. A CSV always has its header, and strings that look like spreadsheet formulas are escaped; the sidecar records how many.

Exit codes

All scrapers use the same exit codes, so one wrapper script can handle any of them:

Codes 3, 4 and 5 are kept apart on purpose. A site that had nothing, a block and a broken parser each call for a different fix.

Access modes: what each site needs

Sites treat automated visitors differently, and the catalog below marks each scraper accordingly. The markers come from dated measurements in each README, so read them as the last known state.

No key, no proxy. 34 of the 65 scrapers ran from an ordinary home connection with nothing configured at the last check. You still need a browser (or nothing at all for the browserless ones), and you should keep request rates modest.

A residential proxy. Some sites serve home connections but refuse datacentre address ranges. The same scraper that works on a laptop then exits with code 3 on a cloud server. Set <SITE>_PROXY to a residential exit: 2Captcha residential proxies work in both formats above, and the Proxy API generates lists for --proxy-file. Selenium drops proxy passwords with a warning, so use Playwright or Puppeteer for authenticated proxies.

Captcha solving. Where a site puts a real captcha in front of the page, such as SHEIN's risk gateway, GeeTest on Avito or DataDome on G2, the repository sends it to 2Captcha when TWOCAPTCHA_KEY is set. Solving is always capped. Flags such as --solve-captcha decide when to solve, and limits such as --max-solves or a fixed number of solves per page cap what a run may spend; the exact flags are in each README. Browser challenges that clear themselves, like Cloudflare's interstitial, get time to settle before a solve is bought or a block is reported.

A hosted browser. 9 scrapers need the 2Captcha Browser API, a Chromium profile hosted by 2Captcha that you connect to over CDP. It keeps cookies and browser state between runs, so a profile that has passed a site's check once usually keeps working. Set <SITE>_CDP_ENDPOINT (or pass --cdp-endpoint) and run as usual. Browser API credentials expire: when you get HTTP 401, create a fresh endpoint as described in the Browser API docs. Selenium can't authenticate a remote endpoint, so use Playwright or Puppeteer.

A fingerprint profile. Some repositories can apply a user agent and related headers from 2Captcha Browser fingerprints to a local browser (--fingerprint, --fp-tags, --fp-country). A few include the client without wiring it into the engines yet; their READMEs say so.

Every README states which mode the site needs and what each paid option adds. If a site needs nothing paid, the README says that too.

Keeping runs comparable

Scrapers for prices and listings often ship a diff tool, such as diff_runs.py in SHEIN and Avito. It compares two complete runs by their unique key and lists what was added, removed or repriced. It refuses comparisons that would mislead: runs of different searches or markets, a sidecar that doesn't match its file, or a run capped by --max-results, where a product may simply have dropped out of the top N. If a repository has no diff tool and you write your own, apply the same checks: compare only runs with the same mode, market, completion status and limits.

Catalog

Access column: No key, no proxy means it ran from a home connection with nothing configured at the last recorded check. 2Captcha Browser API means it needed a hosted browser profile. See README means it depends on where you run it, for example from home it works but from a server it needs a residential exit. Some platforms have several scrapers because their sections are protected differently: TikTok has four, Facebook three.

E-commerce & retail

Classifieds, property & travel

Food & delivery

Apps, social & video

Publishing & Q&A

Jobs & business directories

Finance, markets & fundraising

Sports

All repositories: github.com/2scraper. Missing a site? Open a request.

Site-specific guides

• How to scrape SHEIN: window.gbRawData, the risk gateway and price diffs.
• How to scrape Google Play: per-country ratings and full review histories.
• How to scrape Transfermarkt: AWS WAF and pagination traps.
• How to scrape TipRanks: analyst ratings and price targets.
• How to scrape Lidl: product grids and unit prices.
• How to scrape YouTube: comments and metadata via the Innertube API.
• How to scrape Avito: GeeTest v4 solved from the browser's own IP.

FAQ

Do I need a 2Captcha account to use these scrapers?
Not for most of them. 34 scrapers ran with no key and no proxy at the last recorded check, and the code is MIT-licensed. You need an account only where the README says a site requires captcha solving, a residential proxy or a hosted browser profile, and the README explains what each of those adds.

Why is Playwright recommended over Selenium and Puppeteer?
Playwright is the primary engine in every repository. It gets the live testing, it handles authenticated proxies and remote CDP endpoints, and the canaries use it. Selenium and Puppeteer run the same loop and are kept working, but Selenium can't use authenticated proxies or hosted browsers, and not every repository has verified the other engines end to end.

Can I run several scrapers in parallel?
Yes, as separate processes. Inside one scraper, concurrency is enabled only where pages have their own addresses. Infinite-scroll and cursor-based routes stay sequential because they can't be split between workers. Keep the total request rate modest either way.

Is scraping these sites legal?
The scrapers read publicly visible pages without logging in. Whether a given use is allowed depends on your jurisdiction, the site's terms and what you do with the data, personal data in particular. This is not legal advice: review the terms and your local law, keep request rates low, and don't collect data you have no basis to process.

A scraper exits with code 3 from my server but works on my laptop. What changed?
Your address range. Many sites serve home connections and refuse datacentre ranges. Set a residential proxy in <SITE>_PROXY, or use a Browser API profile if the README recommends one for that site.

How do I know a scraper still works?
Check the canary badge in the README, which shows the last scheduled live run. Run the offline tests after installing, and read .meta.json after every run instead of assuming a non-empty file is complete.

Source code

• Organization: github.com/2scraper, all scrapers under MIT
• Organization profile with the catalog by category: github.com/2scraper/.github

Documentation

• Quick start: API key, first request, first result.
• Browser API docs: CDP endpoint and profile options.
• Proxy API: generating residential proxy lists.
• Browser fingerprints: fingerprint profiles for local browsers.
• Error codes and request limits.
• 2captcha-python: the official Python SDK.