Logo of «2Captcha»To home page
Captcha bypass tutorials

Was this helpful?

How to scrape SHEIN

Gregory Fisher
Gregory Fisher

Technical engineer

SHEIN is a major e-commerce platform in the fashion industry. Scraping its search, category and product pages is harder than loading HTML and querying CSS selectors. The difficulty comes from two factors:

  • Product data is injected into the page's JavaScript state (window.gbRawData for listings) or into schema.org JSON-LD (for product pages), rather than rendered as plain HTML.
  • SHEIN uses a proprietary risk-scoring system that can redirect automated sessions to /risk/challenge (an interactive verification) or /risk/action/limit (a hard rate limit).

In this guide we'll look at how the open-source project 2scraper/shein-scraper works: how to export searches, categories and product pages to JSON or CSV, pass SHEIN's bot check with 2Captcha, and compare runs to track price changes. We'll focus on the Playwright engine, the recommended and most stable one in the repository.

The current implementation targets us.shein.com. Don't assume that other regional domains use the same data structures or endpoints.

What you'll build

One command-line tool that takes a search term, a category or a product URL and returns one row per product. The kind of page is detected from the URL:

Input What it returns Typical use
--query Products for a search term Trend and assortment research
--category Products from a category, as named in SHEIN's own URLs Category price monitoring
--url One product page, read from its schema.org JSON-LD, or any search or category URL Tracking specific products

Each row has sku, category, title, brand, price, currency, original_price, discount_pct, price_source, rating, review_count, store_code, in_stock, is_clearance, quickship, image_url, product_url and scraped_at.

A run-to-run diff tool, diff_runs.py, turns two runs into a list of added, removed and repriced products.

Requirements

  • Python 3.9 or newer.
  • Playwright with Chromium (recommended). Puppeteer (via pyppeteer, no longer maintained) and Selenium versions run the same scraping loop.
  • One virtual environment per engine: their dependencies conflict.
  • For regular use: a 2Captcha API key and a 2Captcha Browser API profile. SHEIN challenges or rate-limits plain local browsers quickly. New to the API? Start with the Quick start guide.

Project setup

bash Copy 
git clone https://github.com/2scraper/shein-scraper.git
cd shein-scraper
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements-playwright.txt
playwright install chromium
cp .env.example .env

Put credentials in .env, never on the command line: TWOCAPTCHA_KEY for the 2Captcha key, SHEIN_CDP_ENDPOINT for a Browser API profile, SHEIN_PROXY for a proxy. Docker users can run docker build -t shein-scraper . to get an image with Playwright and Chromium.

Quick start

bash Copy 
# A search
python3 playwright_scraper.py --query "linen pants" --max-results 60

# A category, as it appears in SHEIN's own URLs, saved as CSV
python3 playwright_scraper.py --category "Women Jeans-c-1934.html" --format csv --out jeans.csv

# A single product page (read from its JSON-LD)
python3 playwright_scraper.py --url "https://us.shein.com/dsbayvkj-p-33704388.html"

# The same search with the other engines
python3 puppeteer_scraper.py --query "summer dress"
python3 selenium_scraper.py  --query "summer dress"   # local browser only

# Compare two runs of the same search
python3 diff_runs.py monday.json tuesday.json

Results go to shein_results.json, with shein_results.json.meta.json beside it. If SHEIN_CDP_ENDPOINT is set in .env, every run goes through your Browser API profile without extra flags.

Defaults: --max-results 30, --format json, --max-scrolls 20, --stall-rounds 4, --scroll-delay 1.5 s, --retries 2.

How the scraper works

Listings come from embedded JSON. Search and category pages carry their products in a JavaScript object, window.gbRawData. The scraper reads it directly, so values like goods_id and price are SHEIN's own. That is also why it needs a browser: the data is built by page scripts, not served as plain HTML.

Product pages come from JSON-LD. A single product page is read from its schema.org ProductGroup or Product markup.

price_source tells you where a price came from: embedded_json (listing data), json_ld (product page) or json_ld_min_variant (the lowest variant price on a product page).

One page, no page numbers. A SHEIN listing is a single infinitely scrolling page with no per-page address. --max-results, --max-scrolls and --stall-rounds take the place of a page count, and there is no --pages or --concurrency: nothing can be handed to a second worker.

Partial runs are flagged. SHEIN reports how many products a listing holds. When the scraper collects fewer, it warns and marks the run partial instead of pretending it got everything.

Anti-bot protection and CAPTCHA on SHEIN

SHEIN sends suspicious visitors to its own risk gateway instead of the page. The scraper recognizes both gates and handles them differently.

1. /risk/challenge: a verification step you can solve. The scraper first clicks the "I am human" checkbox, which costs nothing. If SHEIN escalates, it screenshots the puzzle and sends it to 2Captcha:

  • the 3×3 "select the images matching the icon" grid as a GridTask;
  • the "click the icons in order" puzzle as a CoordinatesTask.

It clicks the answer, and if SHEIN refuses it, repeats with a fresh puzzle, up to --risk-challenge-rounds (default 5) per challenge. --solve-captcha sets when solving runs: off, when-blocked (default) or always.

bash Copy 
python3 playwright_scraper.py --query "summer dress" --max-results 20 \
  --solve-captcha when-blocked --risk-challenge-rounds 5 --max-solves 8

2. /risk/action/limit: a rate limit with nothing to solve. Sending it to a solver would only waste money. The run stops with exit code 3, or, with --rate-limit-cooldown 300, waits five minutes and retries once.

What to expect. In the repository's test on 2026-09-30 with five fresh US Browser API profiles, 3 of 5 passed the challenge, each on its third round: SHEIN usually refuses the first correct answer or two. The other two profiles were refused on every answer, including correct ones, because SHEIN had flagged the profile itself. Only a different profile helps then, and the run exits with code 3 rather than returning bad data.

Spending is capped. --max-solves (default 8) limits paid solves per run; 0 never pays. The number actually spent is saved as solves_spent in the .meta.json.

The recommended setup: 2Captcha Browser API. 2Captcha Browser API gives you a hosted Chromium profile over CDP. Put its endpoint in SHEIN_CDP_ENDPOINT (or pass --cdp-endpoint) and add TWOCAPTCHA_KEY. A profile that has passed the challenge once keeps its cookies and usually scrapes without further challenges. Setup is in the Browser API docs. Selenium can't authenticate a remote endpoint, so use Playwright or Puppeteer for this.

Other options:

Building your own integration? See the API docs, the debugging sandbox and the error codes.

Output files and exit codes

Every run writes the results file and a .meta.json sidecar with status, stop_reason, product count, total_results reported by SHEIN, whether --max-results capped the run, solves spent and a hash of the output file. A run that collects nothing writes neither file, so your previous good data survives. Use --allow-empty if you want an empty file on purpose.

Code Meaning What to do
0 Complete Data written.
1 Crash A bug. Check the traceback and report it.
2 Bad usage Fix the command.
3 Blocked or rate-limited, no rows Switch to a fresh Browser API profile, raise --risk-challenge-rounds, or use --rate-limit-cooldown.
4 No products The page loaded but had no products.
5 Content not obtained A remote service failed, the page never loaded, or SHEIN said the listing has products and none could be parsed (stop_reason parser_found_nothing: the layout has probably changed).
6 Partial Rows were written, but the run didn't finish cleanly. stop_reason says why: blocked, rate_limited, remote_api_error, failed_pages or rejected_rows.

Comparing runs for price monitoring

diff_runs.py compares two complete runs by sku and lists products added, removed and repriced:

bash Copy 
python3 diff_runs.py monday.json tuesday.json --fail-on-change

It refuses comparisons that would mislead you: runs of different searches, markets or --sort values, or a .meta.json that doesn't match its file. A currency change is reported as currency_changed, never as a price change. When a run was capped by --max-results, a product missing from the new run is listed as left_selection (it dropped out of the top N), not as removed. --fail-on-change exits with code 1 on a real price change, so you can use it in a pipeline.

Example output

A real row from the repository's run on 2026-09-30 (sample_output.json has three):

json Copy 
{
  "sku": "458057728",
  "source": "shein.com",
  "category": "Women Mini Dresses",
  "title": "Aloruh Women's Solid Color Sleeveless Mini Dress, Suitable For Beach Vacation,Dresses For Women Summer",
  "brand": "Aloruh",
  "price": 13.03,
  "currency": "USD",
  "price_source": "embedded_json",
  "product_url": "https://us.shein.com/Aloruh-Women-s-Solid-Color-Sleeveless-Mini-Dress-Suitable-For-Beach-Vacation-Dresses-For-Women-Summer-p-458057728.html",
  "image_url": "https://img.ltwebstatic.com/v4/j/pi/2026/05/02/f8/17776852344d34295349aad8f43f4fbd32e5a672d8_thumbnail_405x552.jpg",
  "scraped_at": "2026-09-30T12:56:49Z",
  "original_price": 20.89,
  "discount_pct": 38.0,
  "rating": 4.62,
  "review_count": 1001,
  "store_code": "4534970445",
  "in_stock": true,
  "is_clearance": false,
  "quickship": false
}

sku is SHEIN's goods_id, the same across a product's colors and sizes. original_price and discount_pct are null when there is no discount; rating is null when there are no reviews.

Using the results: deepest discounts

This script reads a run, checks its status and prints the deepest discounts among products with enough reviews to trust the rating:

python Copy 
import json
from pathlib import Path

def best_deals(results_file: str, min_reviews: int = 100, top: int = 10) -> None:
    path = Path(results_file)
    rows = json.loads(path.read_text(encoding="utf-8"))

    meta_path = Path(f"{path}.meta.json")
    if meta_path.exists():
        meta = json.loads(meta_path.read_text(encoding="utf-8"))
        print(f"Run status: {meta.get('status')}, rows: {len(rows)}")

    deals = [
        r for r in rows
        if r.get("discount_pct") and r.get("in_stock")
        and (r.get("review_count") or 0) >= min_reviews
    ]
    deals.sort(key=lambda r: r["discount_pct"], reverse=True)

    for r in deals[:top]:
        print(
            f"-{r['discount_pct']:.0f}%  ${r['price']:.2f} (was ${r['original_price']:.2f})  "
            f"{r.get('rating') or '-'}* / {r['review_count']}  {r['title'][:60]}"
        )

if __name__ == "__main__":
    best_deals("shein_results.json")

Limitations

  • us.shein.com only. Other SHEIN country sites use different locales and haven't been tested.
  • Beyond the first ~20 products is unverified. A listing embeds its first batch of about 20 products, and every live run so far asked for 20 or fewer. The scraper scrolls for more, but whether scrolling loads further products hasn't been confirmed. If it collects fewer than SHEIN reports, the run is marked partial.
  • --sort is recorded but not sent to SHEIN yet. Its URL parameter hasn't been confirmed.
  • Profiles wear out. Browser API credentials eventually stop working; treat an HTTP 401 as the signal to get a fresh endpoint. After roughly ten runs in a row, a profile may be rate-limited for a few minutes.
  • The bot check isn't always solvable. A flagged profile is refused even with correct answers.

FAQ

Is it legal to scrape SHEIN?
The scraper collects publicly visible product and price data without logging in. Whether your use is allowed depends on your jurisdiction and SHEIN's terms of use, so check both and keep request rates modest. This is not legal advice.

Does SHEIN have a public product API?
No. The scraper reads the product data SHEIN already embeds in its pages, window.gbRawData on listings and JSON-LD on product pages.

What CAPTCHA does SHEIN use?
Its own risk gateway: an "I am human" checkbox, a 3×3 image grid matched to an icon, and a "click the icons in order" puzzle. The scraper clicks the checkbox for free and sends the two puzzles to 2Captcha as GridTask and CoordinatesTask.

How much does solving cost per run?
No more than --max-solves paid solves, 8 by default. Passing a challenge took three rounds in tests, and a Browser API profile that passed once usually needs no further solves.

Why does my run end with exit code 3?
Either SHEIN rate-limited you (/risk/action/limit) or it refused every answer for this profile. Wait and use --rate-limit-cooldown for the first case; switch to a fresh Browser API profile for the second.

Conclusion

Scraping SHEIN comes down to two things: reading window.gbRawData instead of the page markup, and treating SHEIN's two gates differently. Solve /risk/challenge with 2Captcha, wait out /risk/action/limit, and use a Browser API profile so one passed challenge keeps working for later runs. Cap spending with --max-solves, let exit codes drive your automation, and use diff_runs.py to turn runs into price-change reports.

Get started: create a 2Captcha account for an API key and try the 2Captcha Browser API.

Source code

Documentation