How to scrape AVITO: A Practical guide

Avito (avito.ru) is Russia's largest classifieds site: phones, cars, real estate, clothing, services — listed by private sellers and shops. For price monitoring or market research, the useful data is right on the public pages: titles, prices and discounts, location, item condition and characteristics, and who is selling.

Getting that data is harder than loading HTML and running CSS selectors over it:

• Avito sits behind its own firewall. It answers too many requests from one address with a «Доступ ограничен: проблема с IP» page. A plain HTTP client never gets past it — requests or curl, even with full browser headers, gets HTTP 429.
• The firewall can ask for a captcha. In our testing it was a GeeTest v4 slide puzzle every time.
• One URL, two renderings. The same listing page sometimes arrives as plain markup and sometimes with an embedded JSON state that carries more fields.
• The obvious price field is the wrong one. On a discounted item, Avito's bare price value is the pre-discount price, not the one the buyer sees.

In this guide we'll walk through the open-source project 2scraper/avito-scraper: how it reaches the pages, where it reads the data from, and how it handles each firewall state. We'll focus on the Playwright engine, which the repository recommends.

What We'll Build

A scraper with three modes:

1. listing — a category or search page, paginated with ?p=N. One row per card.
2. item — individual item pages, either from URLs or straight from a listing run's output.
3. seller — a seller profile plus that seller's listings.

A listing row includes: sku (Avito's item id), title, price, currency, original_price and discount_pct (when the site states them), region_slug, city, location, summary_params, published_at, seller_name, seller_type, seller_id, seller_rating, seller_reviews, promoted, badges, delivery, page and position.

Item mode adds description, params (all characteristics), images, category_path, address, lat/lng, views_total, views_today, seller_since, seller_reply_time and is_active.

What the scraper does not collect: phone numbers (Avito shows them only after login), anything behind login, and the masked IMEI some categories list among item parameters. Private sellers are anonymised by Avito itself, so for them seller_name stays empty and seller_type is private.

Every run also writes a metadata file next to the data. It tells you whether the result is complete, partial or blocked.

Requirements

• Python 3.9+
• Playwright (recommended). The repository also ships pyppeteer and Selenium engines. Install one engine per virtualenv: their dependencies conflict.
• beautifulsoup4 and requests

And, because of the firewall:

• A Russian residential proxy with a sticky session. The browser and the captcha solver must leave from the same IP (more on why below).
• A 2Captcha API key (TWOCAPTCHA_KEY) to solve the GeeTest captcha on the firewall page.
• Optionally, a 2Captcha Browser API endpoint (AVITO_CDP_ENDPOINT), a remote browser over the Chrome DevTools Protocol. This is the option the repository recommends.

Unlike many sites, Avito leaves no free path here. Plain HTTP is refused, and a fresh local browser runs into the captcha almost immediately. The parsing techniques below apply no matter how you connect; getting the pages at all needs a browser, a Russian IP address and, more often than not, a solved captcha.

Project Setup

Clone the repository and install the core dependencies plus the Playwright engine:

git clone https://github.com/2scraper/avito-scraper.git
cd avito-scraper
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

pip install -r requirements.txt
pip install -r requirements-playwright.txt
playwright install chromium

cp .env.example .env

The scraper reads exactly four variables from .env:

Keep credentials in .env, not on the command line: anything on the machine can read argv with ps, and it ends up in shell history and CI logs.

If you use the Browser API, the endpoint and the proxy must be built together, so the remote browser and the captcha solver share one IP. The repository includes a helper that does it in one step and writes both values into .env without printing them:

python3 tools/browser_profile_client.py accounts
python3 tools/browser_profile_client.py connection --account-id <id> --mint-session --write-env

A proxy session lives for a limited number of minutes (-sessTime- in the login). When it expires, the endpoint built on it fails on connect, and you rerun the same command.

python3 env_config.py shows which variables were picked up, without printing their values.

Quick Start

# A category listing, three pages, JSON and CSV
python3 playwright_scraper.py --mode listing \
  --url "https://www.avito.ru/moskva/telefony/mobilnye_telefony/apple-ASgBAgICAkS0wA3OqzmwwQ2I_Dc" \
  --pages 3 --out out/iphones

# A search
python3 playwright_scraper.py --mode listing --url "https://www.avito.ru/moskva?q=iphone+15" --pages 2 --out out/search

# Item pages for what the listing run found
python3 playwright_scraper.py --mode item --from out/iphones.json --max-products 20 --out out/items

# A seller profile and its listings
python3 playwright_scraper.py --mode seller --url "https://www.avito.ru/brands/<seller>" --out out/seller

Add --verbose to see, page by page, what the scraper met and what cleared it.

Each run writes <out>.json and/or <out>.csv, plus two metadata files:

• <out>.meta.json describes the data next to it.
• <out>.latest_attempt.meta.json is written on every ending — success, block, connection error or crash.

The second file matters for automation. A run that finds nothing does not overwrite the previous good output with [], so the attempt file is how a scheduler learns that the latest run failed.

Avito's robots.txt disallows sort and filter parameters (?s=, ?f=, ?radius=, ?view=). The scraper refuses such URLs before fetching anything. Narrow a listing by path instead, the way the site's own URLs do: city → category → subcategory. ?q= search and ?p= pagination are allowed.

How the Scraper Works

The repository separates when to fetch from what a page says:

The flow for one listing page:

1. Build the page URL (?p=N for page 2 onward).
2. Navigate and wait, bounded, until the page is ready. For a listing, "ready" means more than one card is present ([data-marker="item"][data-item-id]), because one match can resolve on an unrelated card-shaped node before the grid renders.
3. Classify the document: content, challenge, captcha, empty, blocked or unknown.
4. Apply the policy for that state (next section).
5. Parse the cards, deduplicate by sku (Avito repeats some promoted cards across pages), move to the next page.
6. Stop at the site's last page, when a page adds no new items, or at --max-products.

The policy lives in page_flow.py as data, so the three engines can't disagree on it:

STATE_POLICY = {
    "content":   {"retry": False, "solve": False, "wait": False, "rotate": False, "blocked": False},
    "challenge": {"retry": False, "solve": False, "wait": True,  "rotate": True,  "blocked": True},
    "captcha":   {"retry": False, "solve": True,  "wait": True,  "rotate": True,  "blocked": True},
    "empty":     {"retry": False, "solve": False, "wait": False, "rotate": False, "blocked": False},
    "blocked":   {"retry": False, "solve": False, "wait": False, "rotate": True,  "blocked": True},
    "unknown":   {"retry": True,  "solve": False, "wait": False, "rotate": False, "blocked": False},
}

An empty listing is treated as a correct answer, not a fault. A blocked page means "try a different address", not "try again".

Why a browser and not requests? The firewall refuses plain HTTP clients outright, and one of its states is a proof-of-work page that only a browser can run. The data itself is in the served HTML, so once a browser gets past the firewall, parsing is ordinary HTML/JSON work.

Data Extraction

Listings: two renderings of the same URL

The same listing URL comes back in one of two forms:

• Plain markup. The grid is in the DOM only: 50 cards per page, each with schema.org microdata (itemprop="price", priceCurrency, name, url).
• Markup plus embedded JSON. The same cards, plus a <script type="mime/invalid" data-mfe-state> block whose catalog.items carry fields the DOM doesn't have: timestamps, the old price of a discounted item, the seller type.

The parser reads both. The JSON wins where it exists, and every row records its source in price_source (state or dom):

def listing_state(html):
    m = re.search(r"<script[^>]*data-mfe-state[^>]*>(.*?)</script>", html, re.S)
    if not m:
        return None
    data = json.loads(m.group(1))
    return data["loaderData"]["data"]["catalog"]     # catalog["items"] is the list

The DOM path reads the card attributes and microdata:

for card in soup.select('[data-marker="item"][data-item-id]'):
    sku = card["data-item-id"]
    title = card.select_one('[data-marker="item-title"]') or card.select_one('[itemprop="name"]')
    price = card.select_one('[itemprop="price"]')            # its `content` attribute
    location = card.select_one('[data-marker="item-location"]')
    params = card.select_one('[data-marker="item-specific-params"]')
    rating = card.select_one('[data-marker="seller-rating/score"]')

A card that the JSON doesn't know about is still taken from the DOM, so a partial JSON block never loses rows.

The price trap

On a discounted item, Avito's top-level priceDetailed.value in the listing JSON (and item.price on the item page) holds the old price. The price the buyer sees sits in the PriceStep "price" component, and the old price is stated separately in the "discount" component:

for step in item["iva"]["PriceStep"]:
    component = step["componentData"]["component"]
    if component == "price":
        price = step["payload"]["priceDetailed"]["value"]       # displayed price
    elif component == "discount":
        original_price = step["payload"]["old_string"]          # e.g. "69 989 ₽"
        discount_pct = step["payload"]["percent"]

On the DOM side, the itemprop="price" microdata already holds the displayed price. The scraper never computes original_price: it is filled only from the site's own figure, so plain-markup rows simply leave it empty.

Results that are not results

Two kinds of cards on a listing page have the same markup as real results but aren't:

• A search with no matches still returns about fifty cards, under a heading «Похожие объявления рядом и в других городах» ("similar listings nearby and in other cities"). The parser cuts the page at that heading. A no-match search ends as an empty result (exit code 4) instead of a complete-looking answer to a different question.
• A «Подобрали для вас» ("picked for you") recommendation carousel can sit inside the results grid with cards of the same data-marker="item" shape. The parser drops any card inside [data-marker="itemsCarousel"]. In the JSON state these cards live in a separate list, so only the DOM path needs the filter.

cards = [c for c in cards if c.find_parent(attrs={"data-marker": "itemsCarousel"}) is None]

Item pages

An item page carries its data in a hydration payload:

window.__staticRouterHydrationData = JSON.parse("…")

It's a JavaScript string literal that contains JSON, so the parser decodes it twice — the literal first, then the JSON inside it — and takes buyerItem from loaderData:

k = html.find("__staticRouterHydrationData")
q = html.find('JSON.parse("', k)
literal, _ = json.JSONDecoder().raw_decode(html, q + len("JSON.parse("))
data = json.loads(literal)
buyer_item = next(v["buyerItem"] for v in data["loaderData"].values()
                  if isinstance(v, dict) and "buyerItem" in v)

The displayed price on the item page is item.formattedPrice.value, again not item.price.

Seller profiles

A seller page (/brands/<slug>) is parsed from the DOM: name, rating, review count, «На Авито с …» tenure, subscribers and the active-listings counter. The profile itself renders only the first 15 listings. To get the rest, the scraper opens the seller's feed, clicks its «Показать все» button and scrolls; every scroll loads 15 more. A run that collects fewer listings than the profile's own counter is reported as partial, not complete.

Working with Avito's Anti-Bot Protection

Avito's protection is its own firewall, «Доступ ограничен: проблема с IP». Its text says why it appears: too many requests from this address. In other words, it is a request-rate decision tied to the IP.

The repository recognises three states of that page:

The third state is not a captcha, and sending it to a solver is pointless. The scraper classifies it as blocked and switches to a new IP address when it can.

Detection is ordered by how much each signal proves:

WALL_TITLE = "Доступ ограничен"
WAIT_TEXT = "подождите немного и обновите страницу"

def page_state(html):
    if WALL_TITLE not in title_of(html):
        return "content" if has_listing_cards(html) else ...
    if re.search(r"<form\b[^>]*\bjs-firewall-form\b", html):
        return "captcha"          # the firewall's captcha form is on the page
    if WAIT_TEXT in html:
        return "blocked"          # rejected solution: switch to a new IP
    return "challenge"            # proof-of-work: let the browser clear it

Two details are worth copying:

• Look for the form element, not a substring. The firewall page's own script mentions .js-firewall-form in a querySelector whether or not the form is on the page.
• Don't test for the word "captcha" in the HTML. The Browser API injects its own …/captcha/… scripts into every page it loads, so a substring test would flag every page as blocked.

Because the firewall reacts to the request rate from one address, the repository refuses --concurrency above 1 and waits 2 s between pages by default (--delay).

Using Proxies

Proxy support is built in. AVITO_PROXY is a single proxy URL:

http://user:pass@host:port

For a 2Captcha residential gateway, the login selects the region and pins the session:

http://{login}-zone-custom-region-RU-session-{id}-sessTime-120:{password}@<gateway-host>:<port>

How the proxy is used depends on how the browser runs:

• Local browser: all browser traffic goes through it. Playwright gets the credentials through its own proxy fields rather than --proxy-server=, so they never land on the browser's command line:

  python Copy

  launch["proxy"] = {"server": "http://host:port", "username": user, "password": password}
  browser = pw.chromium.launch(**launch)

  A gateway login without -session- would rotate the IP on every request, so the scraper pins one session for the run automatically.

• Browser API: the remote browser comes with its own IP address. AVITO_PROXY is then used only by the captcha solver, and it has to be the exact session the endpoint was built with.

When a page stays blocked in a local browser, the scraper starts a new browser with a new IP address (--proxy-block-retries, default 1). It never moves a live session to a new IP: cookies the firewall issued to one address, replayed from another, look more suspicious than either IP alone. New IP addresses come from a --proxy-file pool or, for a 2Captcha gateway, from newly minted sessions. --proxy-rotate per-page uses a new IP address for every page, at the cost of a browser relaunch per page.

Working with CAPTCHA (GeeTest v4)

In our testing, every captcha the firewall served was GeeTest v4 (slide). Here is how the repository solves it.

1. Detect it

The firewall page contains blocks for three captcha types, and the page's script un-hides the one the server picked. wall_variant() returns the first block that isn't display: none. For GeeTest that is #geetest_captcha, and the GeeTest id must also be present.

2. Extract the parameters

GeeTest v4 needs one parameter, the captcha_id. It is hard-coded in the firewall page's inline script:

captcha_id = re.search(r"captchaId\s*=\s*['\"]([0-9a-f]{32})['\"]", html).group(1)

3. Send it to the solver — through the browser's IP

This is the detail that decides success. A token solved from a different IP than the browser's is rejected: Avito answers it with another captcha. So the task is a GeeTestTask with proxy fields, carrying the same sticky-session proxy the browser uses:

task = {
    "type": "GeeTestTask",
    "websiteURL": page_url,
    "version": 4,
    "initParameters": {"captcha_id": captcha_id},
    "proxyType": "http",
    "proxyAddress": proxy_host,
    "proxyPort": proxy_port,
    "proxyLogin": proxy_login,
    "proxyPassword": proxy_password,
}
task_id = requests.post("https://api.2captcha.com/createTask",
                        json={"clientKey": API_KEY, "task": task}).json()["taskId"]

while True:
    time.sleep(5)
    result = requests.post("https://api.2captcha.com/getTaskResult",
                           json={"clientKey": API_KEY, "taskId": task_id}).json()
    if result.get("status") == "ready":
        solution = result["solution"]
        # {captcha_id, lot_number, pass_token, gen_time, captcha_output}
        break

The repository refuses to build a proxyless task by default (solve_geetest_v4 raises an error when there is no proxy), because paying for a token the site will reject is worse than reporting the page as blocked. It also waits up to 240 s for the verdict, longer than the solver may take to give up on a hard puzzle. A shorter timeout abandons the task before you learn why it failed.

4. Submit it through the page's own form

Rather than calling Avito's verify endpoint directly, the scraper puts the solution where the page's own code would and triggers the form's own submit handler. The handler adds the headers the site expects, posts to /web/3/firewallCaptcha/verify and reloads the page, the same path a person's solve takes:

value = json.dumps({"captcha_id": captcha_id, "lot_number": solution["lot_number"],
                    "pass_token": solution["pass_token"], "gen_time": solution["gen_time"],
                    "captcha_output": solution["captcha_output"]})

page.evaluate("""value => {
  const inp = document.querySelector('input[name=captcha-response]');
  const form = document.querySelector('.js-firewall-form');
  inp.value = value;
  form.dispatchEvent(new Event('submit'));
}""", value)

Then it watches what comes back:

• The page's content (cards, an item, a profile, or a genuinely empty listing): the solution was accepted.
• The «подождите…» page: the solution was rejected, so the scraper switches to a new IP address.
• Another captcha: not accepted this time. The page stays blocked and the run reports it.

"The firewall page is gone" alone doesn't count as success: the page passes through intermediate documents while it reloads. While it is navigating, Playwright's page.content() can even throw "page is navigating", so read the HTML inside a try while you wait.

A correct solution doesn't guarantee that Avito accepts the session. In our tests, solutions obtained through the Browser API's IP address were usually accepted. Solutions for a local Chromium were rejected more often, even when solved through the browser's own IP: the site answered with another captcha or the «подождите…» page, and the page came through only after the scraper switched to a new IP address and solved the captcha again. Plan for rotation, and treat each page's outcome as data, not as an exception.

Fallbacks: picture captcha and hCaptcha

The firewall page's code also contains two other captcha types: Avito's own picture captcha and hCaptcha. Neither appeared in our testing, but the scraper handles both in case the server starts assigning them:

• Picture captcha (#inner-captcha): the image is a data: URL, solved with ImageToTextTask. The answer goes into #form-input, then the same form submit.
• hCaptcha (#h-captcha): the sitekey comes from the page's data-sitekey, solved with HCaptchaTask through the browser's IP address. The token goes into #h-captcha-response.

transport_facts.solves[].variant in the run metadata records which type was met (geetest, image or hcaptcha). If you see one of the latter two, that's new behaviour worth checking.

Using 2Captcha Browser API (CDP)

Instead of launching a local Chromium, the Playwright and pyppeteer engines can connect to a remote browser over CDP:

browser = pw.chromium.connect_over_cdp(cdp_endpoint)
context = browser.contexts[0] if browser.contexts else browser.new_context()
page = context.pages[0] if context.pages else context.new_page()

With AVITO_CDP_ENDPOINT set in .env, the CLI does this automatically. The commands from Quick Start don't change.

This is the option the repository recommends for Avito, for practical reasons:

• The profile is persistent. Once it has passed the firewall, its cookies let it through on later requests: in our runs, consecutive listing, item and seller pages came back without hitting the firewall again. A fresh local browser starts from zero every time.
• It clears the proof-of-work variant by itself.
• It brings its own fingerprint, so the scraper sets no user agent.

On connect, the scraper also turns on the remote browser's built-in captcha auto-solve (Captcha.setAutoSolve) and gives it 25 s. On Avito's GeeTest captcha the auto-solve didn't succeed in our tests, so the scraper falls back to the GeeTestTask flow above. That fallback is why AVITO_PROXY must be the session the endpoint was built on.

A few operational facts the code accounts for:

• One live connection per profile. A second one gets 500 profile_locked until the first is released. A cleanly closed run releases it quickly; a killed one can hold it much longer. The driver always closes the browser handle at the end.
• The endpoint expires with its proxy session. The answer is 500 proxy_error on connect; rebuild it with browser_profile_client.py connection --mint-session --write-env.
• Selenium can't use it. Chromedriver has nowhere to put the endpoint's credentials, so the Selenium engine runs a local Chrome only.

Scaling the Scraper

The repository gives you the building blocks. Production orchestration is up to you:

• Parallelism means more IP addresses, not more threads. --concurrency above 1 is refused, because parallel fetches from one address make the firewall appear more often. A Browser API profile also allows only one connection at a time. To go faster, run separate processes, each with its own IP address or profile.

• Keep a delay. --delay defaults to 2 s between pages. The repository makes no claim about a "safe" request rate for Avito, so measure how often the firewall appears on your own traffic.

• Reuse sessions. For periodic monitoring, a persistent Browser API profile that has already passed the firewall beats a fresh session per run.

• Separate a timeout from a refusal. A navigation timeout is retried with the same IP (--retries, --retry-delay). A dead proxy (ERR_CONNECTION_RESET, ERR_PROXY_…) or a blocked page gets a new IP address (--proxy-block-retries).

• Use the exit codes:

  Code	Meaning
  0	Complete
  1	Crash — a bug in the code, not the site or the connection
  2	Bad usage — wrong host or mode, robots-disallowed URL
  3	Blocked — the firewall stayed up after every step
  4	Zero results, including a search with no matches
  5	Connection error — the site was never reached (timeout, dead proxy, profile_locked, proxy_error)
  6	Partial — some pages fetched, then stopped

  3 and 5 call for different fixes: 5 means rebuild the endpoint or retry, not "the firewall won".

• Check completeness. status in <out>.meta.json is complete, partial or failed. transport_facts lists every firewall page met, which step cleared it, every solve with its time, cost and 2Captcha task_id, and every rotation.

• Compare runs safely. diff_runs.py --old A.json --new B.json reports added, removed and changed items by sku. It refuses to compare incomplete runs or runs over a different listing or page count, which would otherwise show up as items "disappearing". It also doesn't report the difference between the two renderings as a price change.

Troubleshooting

Example Output

A listing row (--mode listing). This one comes from the plain-markup rendering, so the JSON-only fields are null; seller name and slug are placeholders:

{
  "source": "avito.ru",
  "scraped_at": "2026-10-06T11:43:07.634024+00:00",
  "url": "https://www.avito.ru/moskva/telefony/iphone_17_256_gb_sim_esim_8212942358",
  "sku": "8212942358",
  "title": "iPhone 17, 256 ГБ, SIM + eSIM",
  "image_url": "https://…img.avito.st/image/1/…",
  "price": 64935.0,
  "currency": "RUB",
  "category": null,
  "price_source": "dom",
  "original_price": null,
  "discount_pct": null,
  "region_slug": "moskva",
  "city": "Москва",
  "location": "Багратионовская, 6–10 мин.",
  "summary_params": "Новый",
  "published_at": null,
  "published_text": null,
  "seller_name": "Продавец 1",
  "seller_type": null,
  "seller_id": "seller1",
  "seller_url": "https://www.avito.ru/brands/seller1",
  "seller_rating": 5.0,
  "seller_reviews": 624,
  "promoted": true,
  "badges": ["Реквизиты проверены", "яПомогаю"],
  "delivery": null,
  "page": 1,
  "position": 1
}

An item row (--mode item, shortened) adds, among others, the full characteristics:

{
  "sku": "7924452716",
  "price": 39990.0,
  "original_price": null,
  "params": {
    "Состояние": "Новое",
    "Производитель": "Apple",
    "Модель": "iPhone 15",
    "Встроенная память": "128 ГБ",
    "SIM-карты": "SIM + eSIM",
    "Цвет": "Розовый"
  },
  "views_total": 4503,
  "seller_type": "company",
  "is_active": true
}

And the run metadata (<out>.meta.json, shortened):

{
  "source": "avito.ru",
  "mode": "listing",
  "status": "complete",
  "stop_reason": "completed",
  "pages_requested": 3,
  "pages_completed": 3,
  "pages_failed": [],
  "products": 147,
  "transport_facts": {
    "over_cdp": true,
    "walls": [],
    "solves": [],
    "rotations": 0
  }
}

Complete Compact Code Example

Below is a single-file version of the repository's core: open a listing in Playwright (locally through a proxy, or over a Browser API CDP endpoint), classify the firewall state, solve GeeTest v4 through the browser's IP address if the firewall shows it, and parse the cards with the price trap, the carousel and the "similar listings" cut-off handled. It leaves out pagination, IP rotation, item and seller pages, and the metadata files. For those, use the repository.

"""Minimal avito.ru listing scraper — the core of 2scraper/avito-scraper in one file.

    pip install playwright beautifulsoup4 requests && playwright install chromium
    export AVITO_PROXY='http://user:pass@host:port'   # Russian residential, sticky session
    export TWOCAPTCHA_KEY='...'                        # only needed if Avito shows the GeeTest captcha
    python3 avito_minimal.py "https://www.avito.ru/moskva?q=iphone+15"

Set AVITO_CDP_ENDPOINT instead of launching a local Chromium to use a remote browser.
"""
import json
import os
import re
import sys
import time
from urllib.parse import urlparse

import requests
from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

WALL_TITLE = "Доступ ограничен"
WAIT_WALL_TEXT = "подождите немного и обновите страницу"
CARD = '[data-marker="item"][data-item-id]'


# ---------------------------------------------------------------- page state
def page_state(html: str) -> str:
    title = re.search(r"<title[^>]*>(.*?)</title>", html, re.S)
    if not title or WALL_TITLE not in title.group(1):
        return "content" if 'data-marker="item"' in html else "unknown"
    if re.search(r"<form\b[^>]*\bjs-firewall-form\b", html):
        return "captcha"            # GeeTest v4 behind the «Продолжить» button
    if WAIT_WALL_TEXT in html:
        return "blocked"            # follows a rejected solution: change IP
    return "challenge"              # proof-of-work, the browser clears it itself


# ------------------------------------------------------------ GeeTest solving
def solve_geetest(api_key: str, captcha_id: str, page_url: str, proxy: str) -> dict:
    p = urlparse(proxy)
    task = {"type": "GeeTestTask", "websiteURL": page_url, "version": 4,
            "initParameters": {"captcha_id": captcha_id},
            # Solve from the browser's own IP, or Avito rejects the token.
            "proxyType": p.scheme, "proxyAddress": p.hostname, "proxyPort": p.port,
            "proxyLogin": p.username, "proxyPassword": p.password}
    r = requests.post("https://api.2captcha.com/createTask",
                      json={"clientKey": api_key, "task": task}, timeout=30).json()
    if r.get("errorId"):
        raise RuntimeError(r.get("errorCode"))
    for _ in range(48):                                    # up to 240 s
        time.sleep(5)
        res = requests.post("https://api.2captcha.com/getTaskResult",
                            json={"clientKey": api_key, "taskId": r["taskId"]}, timeout=30).json()
        if res.get("errorId"):
            raise RuntimeError(res.get("errorCode"))
        if res.get("status") == "ready":
            return res["solution"]
    raise TimeoutError("no solution within 240 s")


SUBMIT_JS = """value => {
  const inp = document.querySelector('input[name=captcha-response]');
  const form = document.querySelector('.js-firewall-form');
  if (!inp || !form) return false;
  inp.value = value;
  form.dispatchEvent(new Event('submit'));   // the page's own handler posts /verify
  return true;
}"""


def content(page) -> str:
    try:
        return page.content()
    except Exception:                                      # the page is mid-reload
        return ""


def pass_wall(page, url: str, proxy: str) -> str:
    html = content(page)
    for _ in range(25):                                    # proof-of-work: just wait
        if page_state(html) != "challenge":
            break
        page.wait_for_timeout(1000)
        html = content(page)
    if page_state(html) != "captcha":
        return html
    key = os.environ.get("TWOCAPTCHA_KEY")
    captcha_id = re.search(r"captchaId\s*=\s*['\"]([0-9a-f]{32})['\"]", html)
    if not (key and proxy and captcha_id):
        return html
    sol = solve_geetest(key, captcha_id.group(1), url, proxy)
    value = json.dumps({k: sol.get(k) for k in
                        ("lot_number", "pass_token", "gen_time", "captcha_output")}
                       | {"captcha_id": captcha_id.group(1)})
    page.evaluate(SUBMIT_JS, value)
    for _ in range(20):                                    # wait for the reload
        page.wait_for_timeout(1500)
        html = content(page) or html
        if page_state(html) in ("content", "blocked"):
            break
    return html


# ------------------------------------------------------------------- parsing
def listing_state(html: str) -> dict:
    m = re.search(r"<script[^>]*data-mfe-state[^>]*>(.*?)</script>", html, re.S)
    if not m:
        return {}
    try:
        catalog = json.loads(m.group(1))["loaderData"]["data"]["catalog"]
    except (ValueError, KeyError, TypeError):
        return {}
    return {str(i["id"]): i for i in catalog.get("items", []) if i.get("type", "item") == "item"}


def prices(item: dict):
    """(displayed price, old price). The bare `priceDetailed` is the PRE-discount
    price; the one the buyer sees is in the `PriceStep` "price" component."""
    shown, old = (item.get("priceDetailed") or {}).get("value"), None
    for step in (item.get("iva") or {}).get("PriceStep") or []:
        component = (step.get("componentData") or {}).get("component")
        if component == "price":
            shown = step["payload"]["priceDetailed"]["value"]
        elif component == "discount":
            digits = re.sub(r"\D", "", step["payload"].get("old_string") or "")  # "69 989 ₽"
            old = float(digits) if digits else None
    return shown, old


def parse_listing(html: str) -> list:
    soup = BeautifulSoup(html, "html.parser")
    state = listing_state(html)
    # A search with no matches still returns cards, under «Похожие объявления…».
    cut = [i for i in (html.find("extraTitle-"), html.find("Похожие объявления")) if i >= 0]
    results = set(re.findall(r'data-item-id="(\d+)"', html[:min(cut)])) if cut else None
    rows, seen = [], set()
    for card in soup.select(CARD):
        if results is not None and card["data-item-id"] not in results:
            continue                                       # substitutes, not results
        if card.find_parent(attrs={"data-marker": "itemsCarousel"}):
            continue                                       # «Подобрали для вас», not results
        sku = card["data-item-id"]
        if sku in seen:
            continue
        seen.add(sku)
        if sku in state:
            item = state[sku]
            price, old = prices(item)
            rows.append({"sku": sku, "title": item.get("title"),
                         "price": price, "original_price": old, "price_source": "state",
                         "url": "https://www.avito.ru" + item.get("urlPath", "").split("?")[0]})
        else:
            title = card.select_one('[itemprop="name"]')
            price = card.select_one('[itemprop="price"]')
            link = card.select_one('a[itemprop="url"], a[data-marker="item-title"]')
            rows.append({"sku": sku,
                         "title": title.get_text(" ", strip=True) if title else None,
                         "price": float(price["content"]) if price and price.get("content") else None,
                         "original_price": None, "price_source": "dom",
                         "url": "https://www.avito.ru" + link["href"].split("?")[0] if link else None})
    return rows


# ---------------------------------------------------------------------- main
def main(url: str) -> int:
    proxy = os.environ.get("AVITO_PROXY")
    cdp = os.environ.get("AVITO_CDP_ENDPOINT")
    with sync_playwright() as pw:
        if cdp:
            browser = pw.chromium.connect_over_cdp(cdp)
            context = browser.contexts[0] if browser.contexts else browser.new_context()
        else:
            launch = {"headless": True}
            if proxy:
                p = urlparse(proxy)
                launch["proxy"] = {"server": f"{p.scheme}://{p.hostname}:{p.port}",
                                   "username": p.username, "password": p.password}
            browser = pw.chromium.launch(**launch)
            context = browser.new_context(locale="ru-RU")
        page = context.pages[0] if context.pages else context.new_page()
        page.goto(url, wait_until="domcontentloaded", timeout=60_000)
        try:
            page.wait_for_function(
                "() => document.title.includes('Доступ ограничен') || "
                "document.querySelectorAll('[data-marker=\"item\"][data-item-id]').length > 1",
                timeout=20_000)
        except Exception:
            pass
        html = pass_wall(page, url, proxy)
        state = page_state(html)
        browser.close()

    if state != "content":
        print(f"blocked: page state is {state!r}", file=sys.stderr)
        return 3
    rows = parse_listing(html)
    print(json.dumps(rows[:5], ensure_ascii=False, indent=2))
    print(f"{len(rows)} rows", file=sys.stderr)
    return 0 if rows else 4


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else
                  "https://www.avito.ru/moskva?q=iphone+15"))

The example returns exit code 3 when the page is still blocked. In production that is the point to start a fresh browser with a different IP address, which is what the repository's fetch_with_rotation does.

Testing and Limitations

The repository ships an offline suite that runs with no network and no browser:

python3 smoke_test.py
python3 -m pytest -q

It checks the deterministic parts against fixtures cut from real pages: page-state classification for all three firewall states, both listing renderings, the price trap, item and seller parsing, solver task construction, proxy handling and output/metadata writing.

Live behaviour is a separate question. Confirmed on the real site:

• Playwright over the Browser API: listing, item and seller modes.
• GeeTest v4 solved with GeeTestTask through the browser's IP address and accepted by the site — but not every time.
• All three firewall states: proof-of-work, GeeTest, and the «подождите…» page after a rejected solution.
• The seller feed read up to the profile's listing counter.

Limitations to keep in mind:

• A local Chromium gets the GeeTest captcha on almost every new session, and its solutions are accepted less reliably than on the Browser API, so a local run often needs a rotation and a second solve.
• The picture captcha and hCaptcha branches are implemented from the firewall page's code but were never served in our testing.
• New live behaviour was only checked on the Playwright engine; pyppeteer and Selenium are covered by the offline suite.
• The firewall's thresholds are not documented anywhere, and the repository doesn't claim any.

Conclusion

Scraping Avito comes down to two things: reading the data where the frontend actually keeps it, and treating the firewall as a normal part of the workflow.

On the data side: listing cards come from the DOM's microdata or, when present, the embedded data-mfe-state JSON. Item pages come from __staticRouterHydrationData. Always take the displayed price, never Avito's bare price field. Recommendation carousels and "similar listings" blocks look like results and aren't.

On the access side, Avito offers no plain-HTTP route. You need a browser with a Russian IP address. The proof-of-work page clears itself, GeeTest v4 needs a solution submitted through the page's own form, and the solution only counts when it comes from the browser's own IP. The «подождите…» page is not a captcha at all, just a signal to change the IP address.

For a one-off pull, a local Playwright run with a sticky residential proxy and a captcha key can work, at the cost of a captcha solution per new session. For regular monitoring, a persistent Browser API profile, run metadata and diff_runs.py together keep a temporary block from showing up in your data as listings disappearing.