How to Scrape YouTube

YouTube is the largest video platform, and scraping its data (comments, video metadata, search results) has its own specifics. Unlike many websites, an effective YouTube scraper does not have to rely on heavy headless browsers to render JavaScript.

Instead, it can read data directly from the same endpoints (Innertube API) that the YouTube frontend itself uses. However, there are important nuances: comments are loaded via a chain of continuation tokens, and the site may issue a challenge check upon suspicious activity.

In this guide, we will explore how to extract data reliably and quickly using the open-source project 2scraper/youtube-scraper, minimizing overhead and properly handling protections.

What we will build

A script capable of working in three modes:

1. Comments (default): Extracting threads and their replies from a specific video.
2. Video: Collecting precise metadata (exact ISO upload date, duration, category, keywords).
3. Search: Getting a list of videos by search query (often used as a source of URLs for subsequent comment scraping).

Extracted data (for comment mode) includes: sku (comment ID), title (video title), text, author_name, like_count, reply_count, published_at_approx, reply_level, parent_id, is_pinned, and creator_hearted.

Important: The repository intentionally replaces author names and comment text in its examples (sample_output), as republishing the words of others requires separate legal justification. However, all metrics, timestamps, and flags are extracted in their original, untouched form.

Requirements

• Python: Version 3.9 or newer.
• Transport: By default (--transport auto), a browser is not required. The script uses standard HTTPS requests, making it very fast.
• Browser engines (optional): Playwright, Selenium, or Puppeteer. Needed only if the site starts refusing access via standard HTTP.
• External services (optional): 2Captcha API key (TWOCAPTCHA_KEY) for solving rare challenge checks.

Project setup

1. Clone the repository and create a virtual environment:

   bash Copy

   git clone https://github.com/2scraper/youtube-scraper.git
   cd youtube-scraper
   python3 -m venv .venv
   source .venv/bin/activate  # For Windows: .venv\Scripts\activate

2. Install base dependencies. Browser libraries are not installed by default, which prevents dependency version conflicts:

   bash Copy

   pip install -r requirements.txt

3. If you plan to use browser mode (e.g., to bypass blocks), install the dependencies for a specific engine separately:

   bash Copy

   pip install -r requirements-playwright.txt
   playwright install chromium

Quick start

The main entry point is youtube_scraper.py. Here are launch examples for each mode:

# 1. Comments (default): 3 pages (60 top comments), sorted by newest, with replies
python3 youtube_scraper.py --url "dQw4w9WgXcQ" --pages 3 --sort newest --replies

# 2. Video metadata (exact ISO upload date format, duration, tags)
python3 youtube_scraper.py --url "dQw4w9WgXcQ" --mode video

# 3. Search: find videos by query (to then feed their URLs into comment mode)
python3 youtube_scraper.py --url "web scraping tutorial" --mode search --pages 3

Note on pagination: In comment mode, a page (--pages) is not a screen, but a data portion. One page equals 20 top-level comments. If the --replies flag is enabled, each reply thread will require an additional request.

How the scraper works: transports

The repository architecture separates execution logic (run_core.py) and drivers. This allows flexible transport selection:

Using auto or http provides a significant performance boost, as it eliminates the overhead of launching a browser, parsing the DOM, and executing JavaScript when it is not strictly necessary.

Data extraction

1. Comment mode (innertube api)

The scraper emulates the POST requests made by the YouTube frontend. It receives the initial batch of comments and uses the continuation token from the response to request subsequent portions.

• Comment replies have their own continuation token.
• The parent_id field is automatically derived from the sku (comment ID), as a reply ID always contains the parent comment ID separated by a dot. This allows reconstructing the tree structure without additional backend logic.

2. Video mode (dual request)

To get full metadata, the scraper makes two requests per video:

1. /next: Gets basic information and rendered figures (views, likes).
2. /player: Extracts the exact ISO upload date, duration in seconds, category, and keyword list (tags), which are absent from the first response.

3. Third-party api limitations (important fact)

The repository explicitly states: services like Scraper API, which simply return rendered page HTML, cannot extract YouTube comments. YouTube does not render comments into the raw HTML; they are loaded via a separate POST request that such services do not emulate. Attempts to use waitFor for comments to appear in such services lead to timeouts (HTTP 408) or validation errors (HTTP 422).

Working with anti-bot protection and captcha

Although YouTube rarely blocks standard viewing requests, mass data collection may trigger protections.

Detection and escalation

When using --transport auto, the scraper analyzes the response. If it is classified as a challenge, it launches a browser (if not already running) and switches to it. It does not switch back to HTTP, as a site that blocked a session once is highly likely to do so again.

Solving captcha via 2captcha

If the --solve-captcha when-blocked parameter is enabled (default) and --twocaptcha-key is provided, the scraper will attempt to solve the task.

• Important limitation: CAPTCHA solving works only in browser transport. A pure HTTP client has no DOM tree to inject a solved token into. Therefore, under --transport http, the scraper will honestly warn about the impossibility of solving and terminate, rather than trying endlessly.

Using proxies

Proxy support is built-in and can be specified via --proxy or a --proxy-file.

Critical rotation detail:
If you use --proxy-rotate per-page, the scraper does not just swap the IP address on the fly. It completely closes the current browser session and creates a new one. This is intentional: if the YouTube bot manager issued cookies for IP address A, and you try to use them from IP address B, it creates a much stronger anomaly signal than having no cookies at all. Rebuilding the session from scratch is safer.

Scaling the scraper

1. Limiting parallelism (--concurrency):
   In comments mode, concurrency higher than 1 is ignored (or forcibly reduced to 1). This is not a bug, but an architectural limitation: a comment page is addressed by a token issued by the previous page. Two workers cannot independently request page 2 without the result of page 1. Parallelism is only effective in video mode, where each video has an independent URL.
2. Safe CSV: When exporting to CSV (--format csv), any value starting with =, +, -, @, or a tab (e.g., an author @username) is automatically escaped with a leading quote character. This prevents formula injection attacks in spreadsheets (CSV Injection).
3. Solve limits: Use --max-solves to limit the budget for CAPTCHA solving in case of mass blocking.

Troubleshooting

Example output

A fragment of real JSON output (author data anonymized per repository policy, metrics are real):

{
  "source": "youtube.com",
  "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ&lc=UgzX1Y2Z3...",
  "sku": "UgzX1Y2Z3...",
  "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
  "text": "[Comment text redacted for privacy]",
  "author_name": "@[channel_handle]",
  "author_is_verified": true,
  "like_count": 318000,
  "like_count_text": "318K",
  "reply_count": 962,
  "published_time_text": "1 year ago",
  "published_at_approx": "2025-09-21T02:10:14Z",
  "published_at_precision": "year",
  "edited": false,
  "reply_level": 0,
  "parent_id": null,
  "is_pinned": true,
  "pinned_by": "Pinned by @[channel_handle]",
  "creator_hearted": true,
  "sort": "top",
  "data_source": "innertube.entity"
}

Complete compact code example

Since the repository provides a ready, optimized CLI, the best integration method is to call its logic. Below is a minimal Python script demonstrating how to programmatically invoke the scraper core:

import subprocess
import json
import sys
import os

def scrape_youtube_comments(video_id: str, pages: int = 2):
    """
    Runs youtube_scraper.py with auto transport (HTTPS with browser fallback).
    Returns a list of comments as Python dictionaries.
    """
    command = [
        sys.executable, "youtube_scraper.py",
        "--url", video_id,
        "--pages", str(pages),
        "--format", "json",
        "--out", "temp_output"
    ]
    
    result = subprocess.run(command, capture_output=True, text=True)
    
    if result.returncode != 0:
        print(f"Execution error: {result.stderr}")
        return []
        
    try:
        with open("temp_output.json", "r", encoding="utf-8") as f:
            data = json.load(f)
            return data.get("items", [])
    except FileNotFoundError:
        print("Result file not found.")
        return []
    finally:
        if os.path.exists("temp_output.json"):
            os.remove("temp_output.json")

if __name__ == "__main__":
    comments = scrape_youtube_comments("dQw4w9WgXcQ", pages=2)
    
    print(f"Extracted {len(comments)} comments.")
    for c in comments[:3]:
        print(f"- [{c[\"like_count\"]} likes] {c[\"author_name\"]}: {c[\"published_time_text\"]}")

Conclusion

Reliable YouTube scraping does not require the constant use of heavy browsers. Understanding that the YouTube frontend communicates via the Innertube API allows the use of fast HTTPS transport (--transport auto), saving resources and time.

Browser engines and 2Captcha integration should be viewed as a safety net in case challenge checks are triggered, not as the primary working method. For monitoring tasks, use video mode for precise metadata and always check the output for signs of blocking via execution metadata.