How to use Scraper API

Scraper API turns a web page into data with a single HTTP request. You send a URL; the service opens it in a browser on its side and returns the HTML as the server sent it, the rendered page as Markdown or a screenshot. A second method runs a Google search and returns the results grouped by type.

There is no Chrome to install and no browser to keep alive, which makes Scraper API a good fit for backend jobs, scheduled tasks and AI agents that need the content of a page.

This guide covers the basics: your first request, reading the response, output formats, waiting for content, long tasks, Google search, errors and a complete Python client.

Scraper API or Browser API?

If a task needs clicks or a logged-in session, use Browser API. For the content of a page, Scraper API is the shorter path.

What you'll need

• A 2Captcha account with a positive balance and your 2Captcha API key: the same key you use for captcha solving.
• Any HTTP client. The examples use curl and Python 3.9 or newer with requests.

Set the API address and your key:

BASE_URL="https://scraper.2captcha.com"
API_KEY="YOUR_API_KEY"

Keep the key out of URLs, repositories and logs; in code, read it from an environment variable.

Scraper API is billed per request; current prices are on the pricing page. Every completed task is billed, including one where the target site is unreachable. A task rejected because of invalid parameters (HTTP 422) is not.

Step 1. Fetch your first page

The synchronous endpoint creates a task and waits for its result in the same request:

curl -X POST "$BASE_URL/tasks/sync" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "scrape",
    "url": "https://example.com",
    "data_format": "raw",
    "format": "json"
  }'

The response:

{
  "status": "success",
  "http_code": 200,
  "headers": {"content-type": "text/html; charset=utf-8"},
  "body": "<!doctype html><html lang=en><head>..."
}

status tells you how the scrape went, http_code is what the target site answered, headers are its response headers, and body is the page.

Parameters go flat in the JSON body, which must not exceed 10,000 bytes. A nested params object is rejected with 422.

Step 2. Read the response correctly

A response carries three different statuses, and they are easy to mix up:

HTTP 200 from the API does not mean the site answered 200. A missing page comes back as HTTP 200, status: "success" and http_code: 404: the task worked, and the site said "not found".

Check a scrape result in this order:

1. status: "error" — the site was unreachable; read error.
2. status: "warn" — the page came back, but a wait condition was not met; read warning.
3. http_code is 400 or higher — the site refused or did not find the page.
4. Only then use body.

Step 3. Choose the output

data_format decides what you get; format decides how it is packed:

raw is the HTML the server sent, not the page after JavaScript. If a page builds its content in the browser, that content is missing from raw. Markdown and screenshots are taken from the rendered page.

So for a page that renders its content with JavaScript, ask for Markdown or a screenshot. Markdown is also a compact input for an LLM:

Example Domain

This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.

[Learn more](https://iana.org/help/example-domains)

Markdown takes longer than HTML. Add "fullPage": true to capture the whole page in a screenshot instead of the visible area.

With format: "raw" you can save the result straight to a file:

curl -X POST "$BASE_URL/tasks/sync" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task_type": "scrape", "url": "https://example.com",
       "data_format": "screenshot", "format": "raw", "timeout": 120}' \
  --output page.png

Check the Content-Type before you use the file. If the site is unreachable, you get a JSON error instead of a PNG. If the task outlives the wait, you get the task ID as text (see Step 5), which is why the example raises the wait to "timeout": 120.

Step 4. Wait for the content you need

waitFor tells the service what to wait for before it captures the page. It takes a JSON object:

{"waitFor": {"element": ".price"}}

Three other forms work the same way:

{"element": "#content", "checkVisible": true}
{"text": "Content loaded"}
{"state": "load"}

The first also requires the element to be visible, the second waits for the text to appear, and the third waits for a load state: load or domcontentloaded.

Keep in mind:

• The wait is capped at 30 seconds. If the condition is not met, the page comes back as it is, with status: "warn" and a warning.
• text only catches text that appears after the page loads. Text already in the HTML never "appears", so the wait runs out. For static content, wait for an element.
• Raise the request's timeout to 120. The default synchronous wait is 60 seconds, and a 30-second wait plus the page load can exceed it.

waitFor sent as a string, even one with JSON inside, fails at once with 422.

Step 5. Long tasks: 408 and async requests

/tasks/sync waits up to timeout seconds (1 to 120, 60 by default). If the task takes longer, you get HTTP 408 with the task ID:

{"error": "timeout", "response_id": "01a17ca9-4164-7bc1-8ff9-725331f56d7b", "status": "pending"}

The task keeps running, so don't send it again. Ask for the result instead, every few seconds:

curl "$BASE_URL/tasks/result/$RESPONSE_ID"

202 means it is still running, 200 returns the result, and 422 returns the error. This endpoint needs no key, so keep response_id values private.

For batches, skip the synchronous wait: POST /tasks/request takes the same parameters, answers at once with 201 and a response_id, and you poll as above. You can also add a webhook_url and the service will notify you when the task finishes; the notification format is in the documentation.

Step 6. Search Google

google_search takes an ordinary Google search URL with a q parameter:

curl -X POST "$BASE_URL/tasks/sync" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "google_search",
    "url": "https://www.google.com/search?q=web+scraping+python&hl=en&gl=us",
    "page_num": 1,
    "format": "json"
  }'

The results come grouped by type:

{
  "status": "success",
  "layout": "organic",
  "real_location": "United States",
  "groups": {
    "organic": {
      "count": 8,
      "items": [
        {
          "url": "https://realpython.com/python-web-scraping-practical-introduction/",
          "title": "A Practical Introduction to Web Scraping in Python",
          "position": 1,
          "page_position": {"page": 1, "position": 1}
        }
      ]
    },
    "video": {"count": 1, "items": ["…"]}
  }
}

• Organic links are in groups.organic.items. Other groups — news, video, places, images, ai_overview and more — appear when Google shows them.
• More results: page_num from 1 to 10, about 10 results per page, numbered across pages. To start lower down, add start=10, 20… to the URL.
• Search type comes from the URL: tbm=nws for news, udm=2 for images, udm=7 for videos, udm=1 for places. Image, video and places searches take longer, so set "timeout": 120 for them.
• Region: put gl in the URL, or pass uule with coordinates such as "40.7128,-74.0060,5000" (radius in metres) or a location name. Google does not always apply a location name and may return results for another country; coordinates are more reliable. Check real_location, but note that it can come back as Unknown even when the region was applied.
• status: "no_results" is not an error: Google simply found nothing.
• Markdown: add "data_format": "markdown" to get the results as one document, for example for an LLM.

Errors

Two failures arrive with HTTP 200, because the task itself completed:

• The site is unreachable: the domain does not exist, the connection or TLS fails, the page does not load in time or the site answers 5xx. The result has status: "error", http_code: 0 (or the 5xx) and an error text. It is JSON even with format: "raw".
• The site answers 404 with an empty body. It also comes back as status: "error", with net::ERR_HTTP_RESPONSE_CODE_FAILURE. A 404 page with content arrives normally, with status: "success" and http_code: 404.

Complete Python example

The client below puts the steps together. It runs a task synchronously, switches to polling on 408, checks the result in the right order and returns HTML, Markdown, PNG bytes or Google results.

"""A small Scraper API client: fetch a page (HTML, Markdown or a screenshot) and run a Google search.

    pip install requests
    export SCRAPER_API_KEY=...        # your 2Captcha API key
    python3 scraper_api_example.py
"""
import base64
import os
import sys
import time

import requests

BASE_URL = "https://scraper.2captcha.com"
API_KEY = os.environ["SCRAPER_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}


class ScraperAPIError(Exception):
    pass


def run_task(task, wait=60, poll_every=3, give_up_after=300):
    """Run a task synchronously; if the sync wait runs out (408), keep polling its result."""
    r = requests.post(f"{BASE_URL}/tasks/sync", headers=HEADERS,
                      json={**task, "timeout": wait}, timeout=wait + 30)
    if r.status_code == 408:                       # the task is still running: do not resubmit it
        response_id = r.json()["response_id"]
        deadline = time.time() + give_up_after
        while time.time() < deadline:
            time.sleep(poll_every)
            r = requests.get(f"{BASE_URL}/tasks/result/{response_id}", timeout=60)
            if r.status_code != 202:
                break
        else:
            raise ScraperAPIError(f"task {response_id} still running after {give_up_after} s")
    if r.status_code == 503:                       # this task type is paused for a while
        raise ScraperAPIError(f"service overloaded, retry after {r.headers.get('Retry-After')} s")
    if r.status_code != 200:                       # 400, 401, 402, 404, 410, 422 …
        raise ScraperAPIError(f"HTTP {r.status_code}: {r.text[:200]}")
    return r


def scrape(url, data_format="raw", wait_for=None, full_page=False):
    """The page as HTML/Markdown text, or PNG bytes for a screenshot."""
    task = {"task_type": "scrape", "url": url, "data_format": data_format, "format": "json"}
    if wait_for:
        task["waitFor"] = wait_for
    if full_page:
        task["fullPage"] = True
    result = run_task(task).json()
    # Check in this order: the site unavailable, a warning, the target's own HTTP status.
    if result["status"] == "error":
        raise ScraperAPIError(f"site unavailable: {result.get('error')}")
    if result["status"] == "warn":
        print(f"warning: {result['warning']}", file=sys.stderr)
    if result["http_code"] >= 400:
        raise ScraperAPIError(f"target answered HTTP {result['http_code']}")
    body = result["body"]
    return base64.b64decode(body) if data_format == "screenshot" else body


def google_search(query, country="us", language="en", pages=1):
    """Organic results as a list of dicts (url, title, description, position)."""
    url = f"https://www.google.com/search?q={requests.utils.quote(query)}&hl={language}&gl={country}"
    result = run_task({"task_type": "google_search", "url": url, "page_num": pages, "format": "json"}).json()
    if result["status"] == "error":
        raise ScraperAPIError(result.get("error", "search failed"))
    if result["status"] == "no_results":
        return []
    print(f"region applied by Google: {result.get('real_location')}", file=sys.stderr)
    return result["groups"].get("organic", {}).get("items", [])


if __name__ == "__main__":
    # 1. The page as Markdown: compact text, for example for an LLM.
    md = scrape("https://example.com", data_format="markdown")
    print(f"first line of the Markdown: {md.splitlines()[0]}")

    # 2. A screenshot of the same page.
    png = scrape("https://example.com", data_format="screenshot")
    with open("page.png", "wb") as f:
        f.write(png)
    is_png = png.startswith(b"\x89PNG\r\n\x1a\n")
    print(f"screenshot: {len(png)} bytes, PNG: {is_png}")

    # 3. Google search, top three organic results.
    for item in google_search("web scraping python")[:3]:
        print(f"{item['position']}. {item['title']} — {item['url']}")

The output:

first line of the Markdown: Example Domain
screenshot: 48429 bytes, PNG: True
region applied by Google: United States
1. A Practical Introduction to Web Scraping in Python — https://realpython.com/python-web-scraping-practical-introduction/
2. Python Web Scraping Tutorial — https://www.geeksforgeeks.org/python/python-web-scraping-tutorial/
3. How to start Web scraping with python? : r/learnpython — https://www.reddit.com/r/learnpython/comments/qzr8ir/how_to_start_web_scraping_with_python/

Common problems

Conclusion

Scraper API covers a common scraping need with one request: send a URL, get HTML, Markdown, a screenshot or grouped Google results. The browser runs on the service side, and your code only has to use the data.

Four habits make it reliable: tell the API's HTTP status, the method's status and the site's http_code apart; ask for Markdown or a screenshot when a page renders with JavaScript; give waitFor an element and the request a 120-second timeout; on 408, fetch the result instead of sending the task again.

Useful links

• Scraper API — https://2captcha.com/scraper/scraper-api
• Scraper API pricing — https://2captcha.com/pricing/scraper-api
• Scraper API documentation — https://2captcha.com/scraper/scraper-api/api
• Browser API — https://2captcha.com/scraper/browser-api
• How to use Browser API with captcha solving — https://2captcha.com/h/how-to-use-browser-api-with-captcha-solving