How to scrape Lidl

Scraping product data from online stores has its own specifics, especially when it comes to large chains like Lidl US. Unlike many other websites, an effective scraper for Lidl should not rely on guessing URL structures or using paid APIs without absolute necessity.

In this guide, we will explore how to reliably and quickly extract data using the open-source project 2scraper/lidl-scraper, minimizing overhead and correctly handling regional pricing specifics.

What we will build

A script capable of working in several modes to get a flat list of products:

1. Search by query: extracting products by keyword (for example, whole milk).
2. Categories and deals: getting a list of products from specific sections or weekly deals.
3. Specific product: collecting precise metadata via a direct link to the product page.

Extracted data includes: sku (product identifier), source, category, title, brand, price, currency, price_source, product_url, image_url, scraped_at, unit_price, unit_price_per, unit_size, original_price, discount_pct, is_weekly_deal, lidl_plus_price, deal_valid_from, deal_valid_until, store_id, and zip_code.

Important: prices on the Lidl website depend on the selected store, which is determined via cookies. Without specifying a store, default prices are returned. The scraper allows you to set a specific store or zip code to get accurate regional prices.

Requirements

• Python: version 3.9 or newer.
• Browser engine: playwright (chromium) is used by default. Selenium and puppeteer (pyppeteer) are also supported for compatibility.
• Environment: it is recommended to use a separate virtual environment for each engine, as their dependencies may conflict.
• Optional: 2Captcha API key (TWOCAPTCHA_KEY) and proxies for scaling, though they are not required for basic usage.

Project setup

1. Clone the repository and create a virtual environment:

   bash Copy

   git clone https://github.com/2scraper/lidl-scraper.git
   cd lidl-scraper
   python3 -m venv .venv
   source .venv/bin/activate  # For Windows: .venv\Scripts\activate

2. Install dependencies for the chosen engine. Do not install multiple engines in one environment:

   bash Copy

   # Primary option (recommended)
   pip install -r requirements-playwright.txt
   playwright install chromium
   
   # Alternative options (only if necessary)
   # pip install -r requirements-selenium.txt
   # pip install -r requirements-puppeteer.txt

3. Copy the environment configuration file:

   bash Copy

   cp .env.example .env

   Leave it empty for a local run without proxies and keys.

Quick start

The main entry point is the playwright_scraper.py script. Here are launch examples for different scenarios:

# 1. Search products by query and save to json
python3 playwright_scraper.py --query "whole milk" --format json --out lidl_results.json

# 2. Search with binding to a specific store (by ID or zip code)
python3 playwright_scraper.py --query "sourdough bread" --store-id US01449 --max-results 20
python3 playwright_scraper.py --query "sourdough bread" --zip 07086

# 3. Sort results (for example, by ascending price)
python3 playwright_scraper.py --query milk --sort price_asc

# 4. Extract data from a category or deals section
python3 playwright_scraper.py --category h/cheese-dairy-eggs/h10095761 --max-results 200
python3 playwright_scraper.py --category aw42-fresh-deals/a10104750

# 5. Scan a specific product page
python3 playwright_scraper.py --url "https://www.lidl.com/p/whole-milk/p11675058"

Note on scrolling: instead of paginated navigation, the website uses dynamic loading. The scraper automatically scrolls the page in portions, clicks the button to load more products, and stops when the result limit is reached or no new data appears.

How the scraper works

The project architecture separates execution logic and parsing, ensuring flexibility and reliability:

• Real data source: the main array of data is extracted from the data-gridbox-impression attribute of each product card, which contains a URL-encoded JSON with clean data about the identifier, name, brand, category, and price.
• Website technology: the site runs on Nuxt.js. It was previously assumed to use Next.js, but tests confirmed that product data on the search page is absent from JSON-LD and is extracted directly from DOM attributes.
• Regional pricing: when using the --store-id or --zip flags, the scraper sets four specific cookies (st, wh, zn, ar), forcing the site to display prices relevant to the chosen region.

Data extraction

1. Search and category mode

The scraper emulates real user behavior: it loads the page, scrolls it, waits for content stabilization, and extracts data from every visible element. Units of measurement and price per unit are calculated based on the visible text next to the main price.

2. Single product mode

When passing a direct link to a product, the scraper navigates to the product page, where data is structured even more thoroughly, including the presence of schema.org Product markup.

3. Output contract

Each result row is strictly typed. The price_source field indicates where the data was taken from (embedded_json or dom), eliminating guesswork. The sku identifier is taken directly from the Lidl website or generated deterministically based on the URL if a direct identifier is unavailable.

Working with engines and proxies

The project supports three engines but has strict technical limitations that are important to know:

Important proxy details:

• The scraper does not support authentication in SOCKS proxies at the chromium level. Use HTTP proxies or proxies without authentication.
• When using a proxy with credentials in selenium, they will be automatically stripped with a warning to avoid silent failures.

Advanced techniques and debugging

For stable operation in automated mode, the scraper provides several powerful flags to help handle network instability or website changes:

• --dump-html: If a parsing error occurs, the scraper will save the raw HTML code of the page to a file. This is an invaluable debugging tool: if Lidl changes the layout or the name of the data-gridbox-impression attribute, you will immediately see what the server returned without needing to launch a browser manually.
• --allow-empty: By default, the scraper protects your data. If a new run returns 0 products (for example, due to a temporary network failure or captcha), it will not overwrite your previous successful result file. This flag disables such protection if you specifically need an empty state.
• --stall-rounds and --max-scrolls: The website uses a virtualized product grid. The scraper scrolls the page in portions. If no new products appear within --stall-rounds (a few cycles by default), the scraper gracefully terminates instead of hanging in an infinite loop.

Integration into automation: exit codes

Understanding exit codes is critical for configuring Cron jobs or CI/CD pipelines. The scraper always returns a clear status:

Every successful or partial run creates a sidecar file

Scaling and limitations

1. No parallelism on a single page: since search results represent a single scrollable page, parallel execution parameters (--concurrency) and per-page proxy rotation do not apply. Parallelism only makes sense when running multiple independent search queries.
2. Safe CSV: when exporting to CSV, any values starting with =, +, -, @, a tab, or a line break are automatically escaped with a leading quote character to prevent formula injection attacks in spreadsheets.
3. No captcha: at the time of the latest tests, the Lidl US website did not use active protections like PerimeterX or Cloudflare for regular search queries, making a local run without additional services quite reliable.

Troubleshooting

Example output

A fragment of real output in JSON format (data obtained during live testing):

{
  "sku": "p11675058",
  "source": "lidl.com",
  "category": "Dairy & Eggs",
  "title": "Lactaid whole milk",
  "brand": "LACTAID",
  "price": 6.38,
  "currency": "USD",
  "price_source": "dom",
  "product_url": "https://www.lidl.com/p/lactaid-whole-milk/p11675058",
  "image_url": "https://www.lidl.com/media/product/0/1/1/6/7/5/0/8/lactaid-whole-milk.jpg",
  "scraped_at": "2026-10-18T14:22:00Z",
  "unit_price": 0.07,
  "unit_price_per": "fl.oz.",
  "unit_size": "96 fl.oz.",
  "store_id": "US01449",
  "zip_code": "07086"
}

Example of programmatic processing

Since the scraper outputs structured JSON, it is easy to integrate into any analytical pipelines. Below is an example script that reads the results, filters only discounted products, and outputs the best deals:

import json
from pathlib import Path

def analyze_lidl_deals(json_file: str):
    path = Path(json_file)
    if not path.exists():
        print(f"File {json_file} not found.")
        return

    with open(path, "r", encoding="utf-8") as f:
        data = json.load(f)

    # Check execution metadata
    meta_file = path.with_suffix(".meta.json")
    if meta_file.exists():
        with open(meta_file, "r", encoding="utf-8") as mf:
            meta = json.load(mf)
            print(f"Collection status: {meta.get('status')} (products collected: {len(data)})")

    # Filter only products with an active discount
    discounted_items = [
        item for item in data 
        if item.get("discount_pct", 0) > 0
    ]

    if not discounted_items:
        print("No discounted products found.")
        return

    # Sort by maximum percentage discount
    discounted_items.sort(key=lambda x: x.get("discount_pct", 0), reverse=True)

    print("\n--- Best discounted offers ---")
    for item in discounted_items[:5]:
        title = item.get("title", "No title")
        brand = item.get("brand", "No brand")
        price = item.get("price", 0.0)
        original = item.get("original_price", 0.0)
        discount = item.get("discount_pct", 0)
        
        print(f"• {brand} {title}")
        print(f"  Price: ${price} (was: ${original}) | Discount: {discount}%")
        print(f"  Link: {item.get('product_url')}\n")

if __name__ == "__main__":
    analyze_lidl_deals("lidl_results.json")

Architectural feature: honest fingerprinting

In the 2scraper family of scrapers, there is a strict rule implemented here as well: the fingerprint_client.py module extracts and applies only the User-Agent from the fingerprint API.

The scraper intentionally does not spoof the browser's timezone or locale, even if the API provides this data. The repository authors adhere to the principle: an honest absence of a signal is better than a guess that could be detected as an anomaly. If you cannot confirm the exact mechanism of applying a specific metric in the target browser, it is not injected into the code to avoid creating a false sense of security and complicating maintenance.

Conclusion

Reliable scraping of data from Lidl US does not require complex protection bypasses or constant paid services. Understanding that the website serves data through specific DOM attributes and manages prices via cookies allows for fast and stable local execution using playwright.

Selenium and puppeteer engines are provided for compatibility, but playwright remains the optimal choice for production. Always specify the region (--store-id or --zip) to get real prices, use exit codes to build reliable automation, and check execution metadata to control the quality of the collected data.