Web Scraping with Python: Tutorial & Examples (2026)

Your requests.get() returns HTML. You parse it with BeautifulSoup. The data isn't there.

The page loaded fine in Chrome. But requests got the shell before JavaScript ran — the content you need loaded afterward. Or your script worked fine for weeks, then started failing silently on some URLs in your list but not others.

Python web scraping starts with retrieving website content and extracting the information you need. For pages that return their data in HTML, Requests and BeautifulSoup are often enough. For JavaScript-rendered content, tools such as Playwright can load the page before extraction.

This tutorial walks through scraping a website with Python, saving the results, handling common errors, and choosing between direct HTTP requests, browser automation, and managed APIs.

As with all data collection: respect each site's terms of service and robots.txt. The examples here cover legitimate use cases — price monitoring, research, internal pipelines, publicly available data.

What is Web Scraping with Python?

Web scraping with Python is the process of retrieving information from websites and extracting specific data using Python code. Requests is commonly used to download HTML, while BeautifulSoup helps locate and parse page elements. When content is loaded through JavaScript, developers may need browser automation or a supported data API.

Typical use cases include collecting publicly available product details, article metadata, research information, and other structured website content.

Set Up Python for Web Scraping

Start with Requests for HTTP retrieval and BeautifulSoup for parsing HTML. Install both packages in your Python environment:

</> Bash

python -m pip install requests beautifulsoup4

A virtual environment is recommended when working on a larger scraping project because it keeps dependencies isolated from other Python applications.

How To Scrape a Website in Python

The following example retrieves quotes and author names from Quotes to Scrape, a practice website designed for learning web scraping. It uses Requests to download the HTML and BeautifulSoup to extract the required elements.

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com/"

response = requests.get(url, timeout=15)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
quotes = []

for item in soup.select(".quote"):
    text = item.select_one(".text")
    author = item.select_one(".author")

    if text and author:
        quotes.append({
            "quote": text.get_text(strip=True),
            "author": author.get_text(strip=True)
        })

for quote in quotes:
    print(quote)

The script downloads the page, identifies each quote using the .quote selector, and extracts the text and author into structured Python dictionaries. From here, the records can be saved to a file or processed by another application.

Technical note: This example is drafted against the practice site's expected structure. Test it before publishing, especially the selectors.

Export Scraped Data to CSV or JSON

CSV is useful for working with tabular information in spreadsheets, while JSON preserves the extracted records as structured data for applications and APIs.

import csv
import json

with open("quotes.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(
        file,
        fieldnames=["quote", "author"]
    )
    writer.writeheader()
    writer.writerows(quotes)

with open("quotes.json", "w", encoding="utf-8") as file:
    json.dump(quotes, file, indent=2, ensure_ascii=False)

Both files contain the records extracted in the preceding example. The CSV output can be opened in a spreadsheet, while the JSON output can be used directly in a downstream Python workflow.

Python Web Scraping in 2026: What's Changed

The Python scraping ecosystem itself is stable. requests, BeautifulSoup, lxml, Playwright, Scrapy — these tools work the same as they did two years ago.

What's changed is the web:

More JavaScript rendering. A larger portion of commercial sites now load their primary content via JavaScript after the initial response. The HTML that requests receives is often just a framework shell — a few containers and a script tag, no actual content.

More dynamic sessions. Sites that previously served data to any request now require cookies, tokens, or interaction sequences before they return useful content.

More infrastructure requirements at scale. Running Python scrapers at volume — hundreds of domains, thousands of URLs — introduces infrastructure complexity that wasn't present at small scale: managing browser servers, handling proxy rotation for IP diversity, debugging failures that appear at scale but not locally.

None of this makes requests obsolete. Most scraping tasks are still Tier 1 or Tier 2 — static HTML or moderately JS-rendered pages at moderate volume. For those, Python's existing tools are exactly right.

The change is that the threshold where you need something beyond a Python script arrives sooner, and when it does, the options are clearer than they used to be.

Which Python Web Scraping Approach Should You Use?

Different websites require different extraction methods. The following framework helps determine when direct HTTP requests are sufficient, when browser rendering is necessary, and when managed execution can reduce infrastructure work.

Most sites are Tier 1. Start there. Only move up when the simpler tool has a specific reason not to work.

Tier 1 — requests + BeautifulSoup Still Works Here

If the site is static HTML, stop here. requests is the right tool.

Check for a JSON API first. Many sites that look like they require scraping actually load data from API endpoints your script can call directly. Browser devtools → Network → filter by fetch or application/json. Direct API calls are faster, cleaner, and more stable than HTML parsing.

For sites where the HTML contains the data:

import requests
from bs4 import BeautifulSoup
from typing import Generator

def scrape_static_site(url: str) -> list[dict]:
headers = {"User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)"}
r = requests.get(url, headers=headers, timeout=15)
r.raise_for_status()

soup = BeautifulSoup(r.text, "html.parser")
results = []

for item in soup.select(".listing-item"):
title_el = item.select_one(".title")
price_el = item.select_one(".price")
if title_el and price_el:
results.append({
"title": title_el.text.strip(),
"price": price_el.text.strip(),
"url": item.get("href", ""),
})

The earlier example demonstrated extraction from a known practice website. The function below shows how the same approach can be adapted into a reusable scraper, with selectors that must be customized to match the target page's HTML.

Tier 2 — How To Scrape JavaScript-Rendered Websites

Content that loads after JavaScript runs requires a real browser. The requests library fetches what the server sends before any JavaScript executes. If your target content is populated by client-side code — React, Vue, Angular, dynamic APIs called from the page — it won't be in that initial response.

Two options, given equal treatment:

Option A: Playwright — local control, full flexibility

import asyncio
from playwright.async_api import async_playwright
from typing import Optional

async def scrape_js_page(url: str, wait_selector: str) -> Optional[str]:
async with async_playwright() as p:
browser = None
try:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto(url, wait_until="domcontentloaded", timeout=30000)
await page.wait_for_selector(wait_selector, timeout=10000)
content = await page.inner_text(wait_selector)
return content
except Exception as e:
print(f"Error fetching {url}: {e}")
return None
finally:
if browser:
await browser.close()

result = asyncio.run(scrape_js_page("https://example.com", ".product-grid"))

Playwright is the right choice when you need direct browser control: custom headers, specific interaction sequences, local debugging, CI pipeline tests. You manage the browser process.

Option B: TinyFish Browser API — managed infrastructure, same Playwright code

Connect to TinyFish's managed browser instead of launching one locally. Your selectors and logic stay identical:

import asyncio
import os
from tinyfish import TinyFish
from playwright.async_api import async_playwright
from typing import Optional

async def scrape_js_managed(url: str, selector: str = "body") -> Optional[str]:
"""Extract JS-rendered content using TinyFish managed browser."""
async with async_playwright() as p:
browser = None
try:
session = TinyFish().browser.sessions.create()
browser = await p.chromium.connect_over_cdp(session.cdp_url)
page = await browser.new_page()
await page.goto(url, wait_until="networkidle", timeout=30000)
content = await page.inner_text(selector)
return content
except Exception as e:
print(f"Error loading {url}: {e}")
return None
finally:
if browser:
await browser.close()

content = asyncio.run(scrape_js_managed("https://example.com", ".product-grid"))

The automation logic remains largely the same, while TinyFish provides managed browser sessions and supporting infrastructure.

For local development, Playwright. For scalable extraction without browser server management, TinyFetch.

For more on when Playwright is the right local choice and where it runs into limits at scale, see Scraping Dynamic Websites: When Playwright Breaks.

Tier 3 — When To Use a Managed Web Scraping API

The gap between a working local script and a reliable production pipeline isn't a code quality problem — it's an infrastructure gap.

At scale, your Python code is correct. What fails is everything around it:

• Browser fleet management: 300MB per Chromium instance, crash recovery, session limits
• Proxy routing: IP reputation, session affinity, geographic distribution
• Failure handling: distinguishing rate limits from detection from transient network errors

Managed web APIs can reduce the work of operating browser infrastructure, maintaining sessions, and integrating execution services. TinyFetch supports content retrieval from known URLs, TinyBrowser provides managed browser sessions, and TinyAgent supports goal-directed tasks. Choose the product based on the workflow rather than sending every extraction through an agent. The infrastructure complexity shifts from your codebase to TinyFish's.

For straightforward extraction with managed infrastructure:

import requests
import os

def managed_fetch(url: str) -> dict:
"""Fetch with TinyFish managed infrastructure."""
response = requests.post(
"https://api.fetch.tinyfish.ai",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"]},
json={"urls": [url], "format": "markdown"},
timeout=60
)
if not response.ok:
return {"url": url, "content": None, "ok": False}
results = response.json().get("results", [])
return {"url": url, "content": results[0].get("text") if results else None, "ok": True}

For goal-directed workflows — multi-step navigation, authenticated access, conditional logic:

import requests
import os
from typing import Optional

def run_agent(url: str, goal: str) -> Optional[dict]:
"""Run a goal-directed extraction agent."""
response = requests.post(
"https://agent.tinyfish.ai/v1/automation/run",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"]},
json={"url": url, "goal": goal},
timeout=120
)
if not response.ok:
return None
data = response.json()
# COMPLETED means the run finished, not necessarily that the goal succeeded
if data.get("status") != "COMPLETED" or data.get("result") is None:
return None
return data["result"]

result = run_agent(
url="https://supplier.example.com",
goal="Find all product listings on the current page. Extract name, SKU, price, and stock status. Return as JSON array."
)

For production pipelines, evaluate managed infrastructure based on successful extraction, task latency, reliability, and total operating cost—not only on the number of requests the system can process.

For running Python extraction at scale across hundreds of URLs simultaneously, see How to Crawl a List of URLs at Scale.

Decision Guide: Which Tool for Which Site

1. Does the website provide a supported API?
   → YES: Use the API directly where permitted.
   → NO: Continue.

2. Is the required information in the initial HTML?
   → YES: Use Requests + BeautifulSoup.
   → NO: Continue.

3. Does the page require JavaScript rendering?
   → YES: Use Playwright or a compatible
          managed browser service.
   → NO: Recheck the source and selectors.

4. Does the task need navigation or decisions?
   → YES: Consider a goal-directed agent.
   → NO: Use direct retrieval or browser extraction.

5. Is browser hosting or session management
   becoming an operational burden?
   → YES: Evaluate managed infrastructure.

Common Python Web Scraping Errors

Web scraping can fail during retrieval, rendering, or extraction. Identifying which step failed helps determine whether the issue requires a different selector, a browser, authentication, or request-level error handling.

Getting Started with TinyFish in Python

Install dependencies and set your API key:

Use TinyFetch when you need content from a rendered page rather than the HTML returned by a direct requests call.

Run a goal-directed extraction:

The API call pattern is close to a standard requests.post(). The structured output removes the parsing step for extraction tasks.

Want a project to try this on? Pick one of these web scraping project ideas with source code, from beginner scrapers to agents that handle logins.

Try TinyFish Free

$8 in allowance available, no credit card. The fastest way to test whether TinyFish fits your workflow.

Related Reading

• The Best Web Scraping Tools in 2026
• How to Scrape Dynamic Websites Without Playwright
• How to Crawl a List of URLs at Scale
• Headless Browser vs AI Agents for Web Automation