How to scrape TRANSFERMARKT

Transfermarkt (transfermarkt.com) is the reference site for football market data: player market values, club squads, transfer fees, contracts and national-team records. For scouting tools, fantasy-football analytics or a research dataset, everything useful is on public pages.

Getting it reliably takes more than loading HTML:

• The site sits behind AWS WAF on CloudFront. Whether you get the page or a captcha depends on the IP address you come from. Most addresses get the page; some never do.
• A default headless browser is refused before the WAF is even asked. Headless Chromium announces itself as HeadlessChrome, and CloudFront answers that with HTTP 403.
• There is no JSON-LD. Every field comes from URL patterns, a little schema.org microdata and the site's own CSS classes.
• Pagination and filters hide surprises. The ranking's own "next page" link sometimes points to a URL that returns 404, and the transfers list silently filters out players valued under EUR 500,000.
• The same data comes in five languages. The German, Russian, Korean and Japanese sites format money and name countries differently.

In this guide we'll walk through the open-source project 2scraper/transfermarkt-scraper: how it reaches the pages, where it reads each field from, and how it handles the WAF. We'll focus on the Playwright engine, which the repository recommends.

What We'll Build

A scraper with four modes:

1. market-values — the most-valuable-players ranking, 25 players per page.
2. club-squad — a club's full squad for a season.
3. transfers — the latest-transfers list.
4. player — one player's full profile.

There are two row types, because a transfer is not a player:

• Player rows (ranking, squad, profile): sku (Transfermarkt's player id), title (name), price and currency (market value in EUR), rank, position, age, nationality, nationalities, club, club_id, shirt_number, and on a profile also full_name, birth_date, birth_place, height_m, agent, league, joined_date, contract_until, market_value_last_update, caps and international_goals.
• Transfer rows: sku (transfer id), player_id, title, position, age, nationality, from_club, from_league, to_club, to_league, price (the fee) and fee_type (disclosed, loan, free or unknown).

Every run also writes a metadata file next to the data. It says whether the result is complete, partial or blocked, and which pages failed.

Requirements

• Python 3.9+
• Playwright (recommended). The repository also ships Selenium and pyppeteer engines and a browserless scraper_api_client.py. Install one engine per virtualenv: their dependencies conflict.
• beautifulsoup4 and requests

External services are optional here. From most addresses the site serves its pages to a local browser with no help. You need more only when your IP address is the one getting the captcha:

• A 2Captcha Browser API endpoint (TRANSFERMARKT_CDP_ENDPOINT): a remote browser over the Chrome DevTools Protocol, with its own IP address and a captcha auto-solver. The repository treats it as the default way past the WAF.
• A proxy (TRANSFERMARKT_PROXY or --proxy-file), when only your IP address is the problem.
• A 2Captcha API key (TWOCAPTCHA_KEY) for the Browser API, the proxy helper and the captcha-solving fallback.

Project Setup

git clone https://github.com/2scraper/transfermarkt-scraper.git
cd transfermarkt-scraper
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium

cp .env.example .env
python3 smoke_test.py            # offline suite: no network, no browser, no keys

The scraper reads four variables from .env:

Keep credentials in .env, not on the command line: anything on the machine can read argv with ps, and it ends up in shell history and CI logs. python3 env_config.py shows which variables were picked up without printing their values.

Leave TRANSFERMARKT_CDP_ENDPOINT empty for now. Whether you need the Browser API at all depends on your IP address, which Quick Start checks first.

Quick Start

First check whether your IP address gets the captcha at all. The repository ships a probe for that:

bash tools/waf_probe.sh 10 market-values

It makes ten plain requests and counts how many were answered with a captcha. If the answer is zero, a local browser is all you need:

# The market-value ranking, three pages, JSON and CSV
python3 playwright_scraper.py --mode market-values --pages 3 --format both --out out/ranking

# A club's squad (Manchester City is club 281)
python3 playwright_scraper.py --mode club-squad --club-id 281 --season 2025 --out out/squad

# The latest transfers, with no market-value floor, in 4 parallel workers
python3 playwright_scraper.py --mode transfers --min-value 0 --pages 6 --concurrency 4 --out out/transfers

# One player's full profile (Erling Haaland)
python3 playwright_scraper.py --mode player --player-id 418560 --out out/player

--mode builds the right URL from --club-id, --season or --player-id, or you pass --url directly. A German, Russian, Korean or Japanese URL works the same way: the language is read from the host.

Once TRANSFERMARKT_CDP_ENDPOINT is set in .env, every run goes through the Browser API, and --concurrency is refused with exit code 2: a Browser API profile takes one connection at a time. For a parallel local run, leave the variable empty or override it for one command: TRANSFERMARKT_CDP_ENDPOINT= python3 playwright_scraper.py ….

Each run writes <out>.json and/or <out>.csv, plus <out>.meta.json. A run that finds nothing does not overwrite the previous good output with [], and the exit code tells a scheduler what happened:

How the Scraper Works

The repository separates how to fetch from what a page says:

The flow for one page:

1. Build the URL (?page=N from page 2 on).
2. Navigate and wait, bounded, until the table is there. "Ready" means more than three player links in the table, because one match can resolve on an unrelated element before the table renders.
3. Classify the page: content, captcha, blocked, empty, not_found or server_error.
4. Apply the policy for that state.
5. Parse the rows, deduplicate by sku, move to the next page.
6. Stop when the listing ends: a page past the end repeats an earlier page, and that is how the end is detected.

The policy lives in page_flow.py as data, so the engines can't disagree on it:

STATE_POLICY = {
    "content":      {"retry": False, "solve": False, "blocked": False},
    "blocked":      {"retry": True,  "solve": False, "blocked": True},
    "captcha":      {"retry": True,  "solve": True,  "blocked": True},
    "empty":        {"retry": False, "solve": False, "blocked": False},
    "not_found":    {"retry": False, "solve": False, "blocked": False},
    "server_error": {"retry": False, "solve": False, "blocked": False},
}

Here retry means "try again from a different IP address". A missing player is the site's real answer, so not_found is neither retried nor paid for. A 5xx is retried from the same IP address and then reported as a server error, never as a block.

Data Extraction

No JSON-LD: three anchors instead

None of the page types this repository covers carries a <script type="application/ld+json"> block. Every field comes from one of three places:

1. URL patterns: /profil/spieler/{id}, /verein/{id}, /jumplist/transfers/spieler/{id}/transfer_id/{id}. They are the most stable hook on the site.
2. schema.org microdata on a player's profile header: itemprop="birthDate", "nationality", "height".
3. The site's own class names (table.items, hauptlink, data-header__label) or the column position, everywhere else.

The ranking table

Ranking, squad and transfers pages all render a table.items. A ranking row has six direct cells: rank, player, age, nationality flags, club and market value. The player cell is a nested table.inline-table with the name link in its first row and the position in the second:

table = soup.select_one("#yw1 table.items") or soup.select_one("table.items")
for tr in table.select("tbody > tr"):
    cells = tr.find_all("td", recursive=False)
    if len(cells) < 6:
        continue                                  # spacer and footer rows
    link = cells[1].select_one('a[href*="/profil/spieler/"]')
    player_id = re.search(r"/profil/spieler/(\d+)", link["href"]).group(1)
    position = cells[1].select("table.inline-table tr")[1].get_text(strip=True)
    flags = [img["title"] for img in cells[3].select("img.flaggenrahmen")]
    club = cells[4].select_one('a[href*="/verein/"]')
    price = parse_market_value(cells[5].get_text())

A player with two nationalities has two flags. The first goes to nationality, all of them to nationalities.

Money

Every value is in euros, on every language site. Only the formatting differs, and a missing value is a dash:

MONEY_RE = re.compile(r"€\s*(\d+(?:[.,]\d+)?)\s*(bn|m|k)?", re.I)

def parse_market_value(text):
    m = MONEY_RE.search(text or "")
    if not m:
        return None                     # "-" means no value published, not 0
    amount = float(m.group(1).replace(",", ""))
    return amount * {"bn": 1e9, "m": 1e6, "k": 1e3}[m.group(2).lower()] if m.group(2) else amount

None matters. In a club squad, young players often have no market value yet. Manchester City's squad in our testing had 43 players, 36 of them with a value. Writing 0 for the other seven would put them in a "worthless" bucket instead of an "unknown" one.

Player profiles

A profile header is header.data-header[itemtype="https://schema.org/Person"]. Birth date, nationality and height come from microdata. Position, the joined date and the contract end come from the header's labelled items, where the label is whatever precedes the first colon:

header = soup.select_one('header.data-header[itemtype="https://schema.org/Person"]')
birth = header.select_one('[itemprop="birthDate"]')
height = header.select_one('[itemprop="height"]')
club = header.select_one("span.data-header__club a")
agent = header.select_one("li.data-header__label a[href*='/berater/']")
labels = {}
for el in header.select("li.data-header__label, span.data-header__label"):
    content = el.select_one("span.data-header__content")
    if content:
        key = el.get_text(" ", strip=True).split(":", 1)[0].strip().lower()
        labels[key] = content.get_text(" ", strip=True)       # "joined", "contract expires", …

The full name and place of birth come from the profile's div.info-table. All dates are normalised to YYYY-MM-DD, which matters on the Korean and Japanese sites, where the year comes first.

The pagination trap

The ranking's own "next page" link sometimes comes in a "gallery" form, …/galerie/<id>/page/10//page/2, and that URL returns 404. In our testing that happened on 3 of 8 runs. A scraper that follows the site's link stops at page 1 in those runs. The plain ?page=N URL served the real pages every time, so the repository ignores a gallery link and builds the URL itself. It logs that it did:

page 1's next-link is the gallery shape … fetching ?page=2 instead

A page past the end of a listing doesn't return 404 either. It shows the last page again. So the scraper detects the end by data, not by markup: a page that adds no new sku ends the listing.

The hidden filter on transfers

The latest-transfers list applies a filter you don't see. Its default view only shows players valued at EUR 500,000 or more (minMarktwert=500000), and on a quiet day that is under one page, with no pager at all. Without the filter the same list ran to several pages and about a hundred transfers in our testing. --min-value 0 removes the floor. The metadata records which filter a run used, so a 20-row result is never mistaken for the whole list.

Five language sites

.de, .world (Russian), .co.kr and .jp have the same DOM and the same ids, just different text. The parser normalises them into one English vocabulary:

• Money formats for millions and thousands are read per site:

  Site	Millions	Thousands
  .com	€220.00m	€500k
  .de	220,00 Mio. €	500 Tsd. €
  .world	220,00 млн €	500 тыс €
  .co.kr	220.00 mil. €	500 k €
  .jp	220.00 m €	500 千 €

• Positions and countries are translated through tables taken from the site itself. The ranking page's filter dropdowns list all 254 countries and 16 positions with the same option values on every host, so pairing them by value gives a complete dictionary.

• Club names are not translated. Their list is open-ended, so a squad scraped from .jp keeps the Japanese club names.

Comparing the same ranking page across the four sites gave the same 25 players with identical values in every field except club.

Working with Transfermarkt's Anti-Bot Protection

The site sits behind AWS WAF on CloudFront. Two things decide whether you get the page.

1. The browser has to look like a browser

A default headless Chromium sends HeadlessChrome in its user agent. In our testing CloudFront answered that with HTTP 403, "The request could not be satisfied", before AWS WAF was involved. The same browser with an ordinary Chrome user agent got the full page:

browser = pw.chromium.launch(headless=True)
ua = (f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      f"(KHTML, like Gecko) Chrome/{browser.version} Safari/537.36")
page = browser.new_context(user_agent=ua).new_page()

The repository does this on every local launch.

2. The IP address decides whether AWS WAF steps in

AWS WAF has two responses, and it names the one it chose in the x-amzn-waf-action response header:

In our testing the captcha depended only on the IP address the request came from. It didn't depend on whether the client was curl, a local browser or a remote one, which page was requested, or how fast requests came:

• Exits in a handful of countries got the captcha on every request: the UAE, Hong Kong, Singapore, Indonesia, Vietnam, Saudi Arabia, the Philippines and China.
• In the other 32 countries tested, about 2% of addresses got it, and an address that got it kept getting it.
• 30 requests in a row from one IP address triggered no captcha at all.

That is why the probe in Quick Start comes first. If your IP address is a clean one, you need nothing else. If it isn't, change the IP address or use the Browser API.

Detection is ordered by how much each signal proves. The response header is the strongest. When an engine can't see headers (Selenium and pyppeteer read the DOM), the page's own markers decide:

WAF_ACTION_HEADER = "x-amzn-waf-action"
AWS_WAF_MARKERS = ("captcha.awswaf.com", "token.awswaf.com", "window.gokuProps")

One detail is worth copying: the Browser API's captcha extension injects its own scripts into every page, and their URLs contain the same vendor names. The repository strips extension-injected <script> tags before matching, or every clean page would look like a challenge.

Using Proxies

Proxy support is built in. --proxy takes one URL and --proxy-file a list:

python3 playwright_scraper.py --mode market-values --pages 2 --proxy-file proxies.txt --proxy-shuffle

A few rules the repository follows:

• Credentials never reach the browser's command line. Playwright gets them through its own proxy fields, so they don't show up in ps:

  python Copy

  browser = pw.chromium.launch(proxy={"server": "http://host:port",
                                      "username": user, "password": password})

• One login, several IP addresses. For a 2Captcha residential gateway, --proxy-sessions N builds N sticky sessions from one login (a different -session- id each), and --concurrency gives each worker its own IP address.

• A blocked page moves to a new browser on a new IP address, never the same session on a new IP address: cookies issued to one address and replayed from another look worse than either alone.

• A refused login is named at once. A browser never tells you that the proxy rejected its credentials. It just waits out the page-load timeout. Before the first page, the repository sends one CONNECT to the proxy. A 407 Proxy Authentication Required stops the run within a second or two with stop_reason: proxy_auth_refused, instead of three timed-out attempts.

Selenium can't send proxy credentials at all. The repository strips them and warns, so with Selenium you need a proxy that authorises your machine by IP.

Working with CAPTCHA (AWS WAF)

The only challenge this site served in our testing was AWS WAF. Here is how the repository handles it.

1. Detect it

The x-amzn-waf-action header names the action. On the page itself, the captcha (as opposed to the silent challenge) loads a captcha.js from *.awswaf.com:

WAF_SCRIPT_RE = re.compile(r"<script[^>]+src=[\"']([^\"']*\.awswaf\.com/[^\"']*?(challenge|captcha)\.js)")
scripts = {kind: src for src, kind in WAF_SCRIPT_RE.findall(html)}
has_widget = "captcha" in scripts

A challenge page has no widget, so there is nothing to solve. The scraper gives the browser up to 15 s to run the check by itself and never pays for it.

2. Extract the parameters

Everything a solver needs is printed into the page as window.gokuProps:

props = json.loads(re.search(r"window\.gokuProps\s*=\s*(\{.*?\})\s*;", html, re.S).group(1))
website_key, iv, context = props["key"], props["iv"], props["context"]

If any of the three is missing, the repository refuses to build a task. The API accepts incomplete tasks without complaint, so a partial read would buy a solution that can't work.

3. Send it to the solver — from the browser's IP

2Captcha takes this captcha as AmazonTask. The solution is tied to the IP address it was solved from, so the task carries the same proxy the browser uses:

task = {
    "type": "AmazonTask",                       # AmazonTaskProxyless: no proxy
    "websiteURL": page_url,
    "websiteKey": website_key,
    "iv": iv,
    "context": context,
    "challengeScript": scripts.get("challenge"),
    "captchaScript": scripts.get("captcha"),
    "proxyType": "http", "proxyAddress": host, "proxyPort": port,
    "proxyLogin": login, "proxyPassword": password,
}

Without a proxy the solver works from its own address. In our testing that came back as existing_token: 2Captcha's own IP address wasn't challenged, so nothing was solved.

4. Use the solution

The solution carries a captcha_voucher, and the voucher is not the token. Set as the aws-waf-token cookie, it was refused. The repository exchanges it the same way the widget does, at /voucher next to the page's challenge.js, and sets the token it gets back:

voucher_url = scripts["challenge"].rsplit("/", 1)[0] + "/voucher"
token = requests.post(voucher_url,
                      data=json.dumps({"captcha_voucher": voucher, "existing_token": existing}),
                      headers={"Content-Type": "text/plain;charset=UTF-8"},
                      proxies=proxies, timeout=30).json()["token"]
page.context.add_cookies([{"name": "aws-waf-token", "value": token, "url": page_url}])
page.reload()

This is the fallback, and it is not proven on this site. On IP addresses that got the captcha on every request, a voucher came back in about 10 s in 4 of 6 tries, but the page then served the captcha again. The cause wasn't established. In a local browser the repository calls the solver as soon as it meets the captcha. Over the Browser API it calls it only if the captcha is still there after the browser's own auto-solver reports success.

Other challenges: wired in, not met

The repository's detector also recognises Cloudflare, DataDome, PerimeterX/HUMAN, reCAPTCHA and hCaptcha by their markers on the page. None of them appeared on this site in our testing. A page that shows one is reported as blocked or as a captcha rather than parsed as an empty result. For reCAPTCHA v2 and v3 the solver path is wired in as well (RecaptchaV2TaskProxyless, RecaptchaV3TaskProxyless); for the others, detection is all there is.

Using 2Captcha Browser API (CDP)

The Browser API is a remote Chromium you connect to over CDP. It comes with its own IP address and a persistent profile, so it answers both questions at once: what the browser looks like and where it comes from.

The repository's helper asks 2Captcha to build the endpoint and writes it into .env without printing it:

python3 tools/browser_profile_client.py accounts                # one row per account
python3 tools/browser_profile_client.py use --account-id <id> --write-env

After that, every run goes through the remote browser. Its auto-solve extension handles the AWS WAF captcha:

browser = pw.chromium.connect_over_cdp(endpoint, timeout=30000)
context = browser.contexts[0]
page = context.pages[0] if context.pages else context.new_page()
cdp = context.new_cdp_session(page)
cdp.send("Captcha.setAutoSolve", {"autoSolve": True, "options": [{"type": "*"}]})

In our testing the auto-solver cleared 5 of 5 captchas it met, in 30 to 63 s each. With auto-solve switched off (--no-autosolve), the same endpoint stayed blocked through 8 of 8 reloads. On the IP addresses that got the captcha on every request, though, it reported success and the page served the captcha again, just as with the solver. There, the reliable fix is a different IP address.

Three things to know:

• Don't reload while it solves. The solver navigates the page itself once it has the token. A reload issued at the same moment fails with net::ERR_ABORTED. The repository waits for the table instead.
• One live connection per profile. A second connection to a busy profile is refused with HTTP 500 and profile_locked. For parallel runs, use one profile per run. The repository refuses --concurrency over --cdp-endpoint for that reason.
• Refusals are worth one retry. The service names its refusal in the 500's body (profile_locked, proxy_timeout, proxy_error, fingerprint_error). These are immediate, so the repository retries them after a few seconds and names the word in its hint if they persist.

Without a browser: Scraper API

scraper_api_client.py fetches the same pages through 2Captcha's Scraper API, with no local browser. It uses the same parser and the same page states. In our testing it returned the same transfers rows as the browser engines. A page can take from a few seconds to over a minute, so the client takes a --timeout.

Scaling the Scraper

• Parallel pages: transfers only. The transfers list is addressable with a plain ?page=N, so --concurrency N fetches pages in parallel. Page 1 is always fetched alone, because it decides whether the rest can be addressed directly. Each worker owns its browser and one proxy IP address for its lifetime, so no locks are needed.
• More workers need more IP addresses. Without a proxy pool, N workers send N times the traffic from one address. The repository warns about this.
• Browser API in parallel means several profiles, one run each.
• Retries are bounded and named. --retries counts attempts, not re-tries, and the metadata lists failed pages by number.
• Delays. --delay sets the pause between pages. Don't invent a safe rate for the site: in our testing the rate itself didn't trigger the captcha, but that is a fact about the days and addresses we tested.

Troubleshooting

Example Output

A ranking row:

{
  "source": "www.transfermarkt.com",
  "url": "https://www.transfermarkt.com/x/profil/spieler/937958",
  "sku": "937958",
  "title": "Lamine Yamal",
  "price": 220000000.0,
  "currency": "EUR",
  "price_source": "listing",
  "category": "market-values",
  "rank": 1,
  "page": 1,
  "position": "Right Winger",
  "age": 19,
  "nationality": "Spain",
  "nationalities": ["Spain"],
  "club": "FC Barcelona",
  "club_id": "131"
}

A player profile adds the header fields:

{
  "sku": "418560",
  "title": "Erling Haaland",
  "price": 220000000.0,
  "price_source": "detail",
  "category": "player",
  "position": "Centre-Forward",
  "shirt_number": 9,
  "club": "Manchester City",
  "full_name": "Erling Braut Håland",
  "birth_date": "2000-07-21",
  "birth_place": "Leeds",
  "height_m": 1.95,
  "league": "Premier League",
  "league_id": "GB1",
  "joined_date": "2022-07-01",
  "contract_until": "2034-06-30",
  "market_value_last_update": "2026-07-22",
  "caps": 59,
  "international_goals": 65
}

A transfer row:

{
  "url": "https://www.transfermarkt.com/jumplist/transfers/spieler/1072321/transfer_id/6656507",
  "sku": "6656507",
  "player_id": "1072321",
  "title": "Moisés Paniagua",
  "price": 2000000.0,
  "currency": "EUR",
  "category": "latest-transfers",
  "position": "Left Winger",
  "age": 19,
  "nationality": "Bolivia",
  "from_club": "Club Always Ready",
  "to_club": "Wydad Casablanca",
  "to_league": "Botola Pro Inwi",
  "fee_type": "disclosed"
}

And the metadata file next to the ranking:

{
  "mode": "market-values",
  "status": "complete",
  "stop_reason": "completed",
  "pages_requested": 3,
  "pages_completed": 3,
  "pages_failed": [],
  "products": 75,
  "start_url": "https://www.transfermarkt.com/spieler-statistik/wertvollstespieler/marktwertetop"
}

Complete Compact Code Example

The script below condenses the repository's logic for one mode, the English market-value ranking, into one file. It covers the ordinary user agent, the two AWS WAF actions, the readiness wait, row parsing and the end-of-listing check. It runs a local Chromium, or 2Captcha's Browser API when TRANSFERMARKT_CDP_ENDPOINT is set. For other modes, other languages, proxies and the solver fallback, use the repository.

"""Compact Transfermarkt market-value scraper.

    pip install playwright beautifulsoup4 && playwright install chromium
    python3 transfermarkt_minimal.py --pages 3 --out ranking.json
"""
import argparse
import json
import os
import re
import sys
import time

from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

RANKING_URL = "https://www.transfermarkt.com/spieler-statistik/wertvollstespieler/marktwertetop"
ROW_SELECTOR = '#yw1 table.items tbody > tr a[href*="/profil/spieler/"]'
MONEY_RE = re.compile(r"€\s*(\d+(?:[.,]\d+)?)\s*(bn|m|k)?", re.I)
SUFFIX = {"bn": 1_000_000_000, "m": 1_000_000, "k": 1_000}
GOKU_RE = re.compile(r"window\.gokuProps\s*=\s*(\{.*?\})\s*;", re.S)


# ------------------------------------------------------------------ page state
def waf_state(html, waf_header):
    """'captcha' (a puzzle widget), 'challenge' (a silent JS check) or None."""
    if waf_header in ("captcha", "challenge"):
        return waf_header
    if html and GOKU_RE.search(html):
        return "captcha" if re.search(r"\.awswaf\.com/[^\"']*captcha\.js", html) else "challenge"
    return None


def rows_on_page(page):
    return page.locator(ROW_SELECTOR).count()


def wait_for_rows(page, timeout_s, minimum=4):
    """More than three player links, not one: a single match can resolve on
    an unrelated element before the table renders."""
    deadline = time.time() + timeout_s
    while time.time() < deadline:
        if rows_on_page(page) >= minimum:
            return True
        page.wait_for_timeout(500)
    return False


# --------------------------------------------------------------------- parsing
def parse_money(text):
    m = MONEY_RE.search(text or "")
    if not m:
        return None                      # "-" means no value published, not 0
    amount = float(m.group(1).replace(",", ""))
    return amount * SUFFIX[m.group(2).lower()] if m.group(2) else amount


def parse_ranking(html, page_num):
    soup = BeautifulSoup(html, "html.parser")
    table = soup.select_one("#yw1 table.items") or soup.select_one("table.items")
    rows = []
    for i, tr in enumerate(table.select("tbody > tr") if table else []):
        cells = tr.find_all("td", recursive=False)
        if len(cells) < 6:
            continue                     # footer and spacer rows
        link = cells[1].select_one('a[href*="/profil/spieler/"]')
        if not link:
            continue
        player_id = re.search(r"/profil/spieler/(\d+)", link["href"]).group(1)
        inline = cells[1].select("table.inline-table tr")
        club = cells[4].select_one('a[href*="/verein/"]')
        flags = [img.get("title") for img in cells[3].select("img.flaggenrahmen") if img.get("title")]
        price = parse_money(cells[5].get_text())
        rows.append({
            "sku": player_id,
            "title": link.get_text(strip=True),
            "url": f"https://www.transfermarkt.com/x/profil/spieler/{player_id}",
            "rank": int(cells[0].get_text(strip=True) or 0) or None,
            "position": inline[1].get_text(strip=True) if len(inline) >= 2 else None,
            "age": int(cells[2].get_text(strip=True)) if cells[2].get_text(strip=True).isdigit() else None,
            "nationality": flags[0] if flags else None,
            "nationalities": flags or None,
            "club": (club.get("title") or club.get_text(strip=True)) if club else None,
            "club_id": re.search(r"/verein/(\d+)", club["href"]).group(1) if club else None,
            "price": price,
            "currency": "EUR" if price is not None else None,
            "page": page_num,
            "row_index": i,
        })
    return rows


# ------------------------------------------------------------------- browser
def open_browser(pw):
    endpoint = os.environ.get("TRANSFERMARKT_CDP_ENDPOINT")
    if not endpoint:
        browser = pw.chromium.launch(headless=True)
        # Headless Chromium announces itself as "HeadlessChrome", and CloudFront
        # answers that with a 403 before AWS WAF is even asked.
        ua = (f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
              f"(KHTML, like Gecko) Chrome/{browser.version} Safari/537.36")
        return browser, browser.new_context(user_agent=ua).new_page(), False
    browser = pw.chromium.connect_over_cdp(endpoint, timeout=30000)
    context = browser.contexts[0] if browser.contexts else browser.new_context()
    page = context.pages[0] if context.pages else context.new_page()
    try:
        cdp = context.new_cdp_session(page)
        cdp.send("Captcha.setAutoSolve", {"autoSolve": True, "options": [{"type": "*"}]})
    except Exception as e:           # a plain CDP browser has no Captcha domain
        print(f"auto-solve not available: {e}", file=sys.stderr)
    return browser, page, True


def fetch(page, url, remote):
    """Load one page and return its HTML once the table is there, or None."""
    response = page.goto(url, wait_until="domcontentloaded", timeout=60000)
    waf = waf_state(page.content(), response.headers.get("x-amzn-waf-action") if response else None)
    if waf == "challenge":
        wait_for_rows(page, 15)      # the browser runs the check by itself, ~1 s
    elif waf == "captcha":
        if not remote:
            print("AWS WAF captcha on this IP address. Use another IP or the Browser API.",
                  file=sys.stderr)
            return None
        wait_for_rows(page, 90)      # the Browser API solves it and reloads the page
    return page.content() if wait_for_rows(page, 20) else None


# ---------------------------------------------------------------------- main
def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--pages", type=int, default=1)
    ap.add_argument("--out", default="ranking.json")
    args = ap.parse_args()

    rows, seen = [], set()
    with sync_playwright() as pw:
        browser, page, remote = open_browser(pw)
        try:
            for n in range(1, args.pages + 1):
                url = RANKING_URL if n == 1 else f"{RANKING_URL}?page={n}"
                html = fetch(page, url, remote)
                if html is None:
                    print(f"page {n}: no table — stopping", file=sys.stderr)
                    break
                new = [r for r in parse_ranking(html, n) if r["sku"] not in seen]
                if not new:              # a page past the end repeats an earlier one
                    break
                seen.update(r["sku"] for r in new)
                rows += new
                print(f"page {n}: {len(new)} rows")
                time.sleep(2)
        finally:
            browser.close()

    if not rows:
        print("0 rows — nothing written", file=sys.stderr)
        sys.exit(3)
    with open(args.out, "w", encoding="utf-8") as f:
        json.dump(rows, f, ensure_ascii=False, indent=2)
    print(f"saved {len(rows)} rows -> {args.out}")


if __name__ == "__main__":
    main()

Testing and Limitations

What was checked:

• The compact script, three pages through a local Chromium and three through the Browser API: 75 rows each time, identical in every field to the repository's own rows for the same pages. Without the ordinary user agent, the local run got HTTP 403.
• The captcha code, on a real captcha page. Through a proxy IP address that gets the captcha on every request, the page came back as HTTP 405 from CloudFront with x-amzn-waf-action: captcha and the title "Human Verification". The detection, gokuProps and script snippets above read it the same way as the repository's own detector, and the compact script stopped with its captcha message. Nothing was sent to the solver.
• The repository, end to end from a fresh clone: all four modes on all three browser engines and the Scraper API, the five language sites, a proxy pool of four IP addresses, the Browser API with parallel profiles, and the Docker image. Rows from different engines were compared with the repository's diff_runs.py: nothing added, removed or changed.
• The offline suite that ships with the repository: no network, no browser, no keys.

Limitations:

• On IP addresses that get the captcha on every request, nothing cleared it. The solver returned a voucher and the Browser API's auto-solver reported success, but the page served the captcha again. Elsewhere the auto-solver cleared every captcha it met. The solver fallback has not been proven to clear the captcha on this site.
• The compact script's Browser API captcha branch wasn't exercised. The Browser API profiles it ran on got no captcha, so the wait for the auto-solver has not met one.
• The country list is a snapshot. Which addresses AWS WAF challenges is the site's decision and can change. Run the probe from your own address.
• Club names are not translated on the language sites, and the "loan" and "free transfer" labels on non-English transfers pages are not recognised yet: those rows get fee_type: "unknown".

Conclusion

We walked through a Transfermarkt scraper that collects market-value rankings, club squads, the latest transfers and full player profiles on five language sites, with run metadata that tells a complete result from a partial or blocked one.

The site's data is plain HTML with stable URL ids, so the parsing is ordinary table work once you know its traps: no JSON-LD, money that needs per-language parsing and null for "no value", a pagination link that sometimes returns 404, and a transfers list that hides cheaper players by default.

Getting the pages depends on where you come from. Use an ordinary user agent, and probe your IP address first. From most addresses a local browser is enough. From an address AWS WAF challenges, use the 2Captcha Browser API, whose auto-solver cleared the captcha in our testing, or change the IP address with a proxy: on an IP address that gets the captcha on every request, a different IP address is the fix. The captcha-solving API is wired in as a fallback.