Fetch Multiple URLs in Python: Async & Batch Guide

Fetching data from multiple URLs is a common task in web scraping, research, and data-processing pipelines. Python can retrieve pages sequentially, but asynchronous requests allow multiple network operations to run concurrently, improving efficiency when workloads spend significant time waiting for responses.

This guide covers how to fetch multiple URLs with Python, manage concurrency, handle failures, and choose the right tool for static pages, JavaScript-rendered websites, and authenticated workflows.

What's in your list → which tool:

1. All static HTML, no strict automation requirements → requests + httpx (fastest, cheapest)
2. JavaScript-rendered content, no strict automation requirements → Playwright or Crawlee
3. Mixed list with some protected sites → Playwright + proxy rotation
4. Protected or authenticated URLs at scale → TinyAgent
5. Massive volume (100K+) of public pages → Scrapy

Fetch Multiple URLs Concurrently in Python

HTTPX is an HTTP client that supports asynchronous requests. Together with asyncio, it lets Python fetch several URLs concurrently while limiting how many requests are active at once.

Install HTTPX:

python -m pip install httpx

</> Python:

import asyncio

import json

import httpx

urls = [

]

async def fetch_url(client, url, semaphore):

async def main():

if name == "main":

The semaphore allows up to five requests to be active concurrently. HTTPX handles the network operations, while asyncio.gather() collects the results. Each record contains the source URL, response status, retrieved HTML, or a description of the failure.

The example also saves the results as JSON, making it easier to inspect the retrieved pages or identify requests that need another attempt.

Handle Failed Requests, Timeouts, and Rate Limits

Bulk fetching should not stop entirely because one URL fails. Record unsuccessful requests, distinguish temporary network problems from permanent HTTP errors, and retry only when another attempt is appropriate.

For rate-limited requests, respect the server's Retry-After instructions and reduce request frequency. Use bounded retries and backoff for temporary failures rather than repeatedly sending requests without delay.

The Tool That Fits the List

Static HTML at Scale Requests, HTTPX, and Asyncio

For smaller URL lists, synchronous Requests calls may be sufficient. For larger lists dominated by network waiting time, asynchronous clients such as HTTPX or aiohttp allow multiple requests to remain in progress concurrently. AsyncIO coordinates asynchronous tasks, but synchronous Requests calls do not automatically become non-blocking simply because they are placed inside an asynchronous function.

For 100K+ URLs, Scrapy's built-in scheduler, downloader middleware, and item pipeline make more sense—it handles deduplication, retry logic, and output formatting at Scrapy's architecture level.

Where this breaks down: Any URL that requires JavaScript execution. If the page shows a loading spinner and populates content after load, requests returns the spinner HTML, not the content. In these scenarios, exploring modern AI web scraping tools can offer a more robust solution for handling complex content automatically.

JavaScript Content: Playwright with Batching

For lists where content loads via JavaScript—React SPAs, infinite scroll, dynamic filtering, price tables that render after an API call—you need a real browser.

Keep concurrency low (3–8 pages) when running locally—each headless Chromium instance consumes 100–300MB. For larger lists, cloud browser infrastructure (Browserless, Browserbase) handles the browser pool so you're not resource-limited on your machine.

When processing many dynamic pages, reuse browser instances where practical and limit concurrent pages based on available memory and the target websites' requirements. Managed browser infrastructure becomes relevant when local execution, session management, or browser capacity is difficult to maintain.

Where this breaks down: Sites with strict automation requirements at the network and behavioral level. JavaScript-level automation handling helps at low volume; at scale, sites with enterprise-grade access requirements become harder to handle reliably.

Sites with Strict Requirements or Authenticated Access: TinyFish

This is where simple HTTP requests stop being sufficient. Your list includes:

• Product pages that return different content to automation than to browsers
• Pricing pages that require login using your own authorized account
• Sites with strict automation requirements that affect reliability at scale
• Authenticated portals where each URL requires an authorized session

For these, maintaining a Playwright-based crawler means:

• Managing automation configuration that needs ongoing updates as site requirements evolve
• Building session management for authenticated URLs
• Handling multi-step login flows and session state
• Debugging failures that change based on site configurations you don't control

AI web agents handle this at the infrastructure level. You pass a URL and a goal; the agent handles rendering, infrastructure-level request handling, and authentication for sites where you have authorized access.

TinyFish standard accounts support two concurrent Agent runs, with higher limits available through Enterprise. For URLs that only need content retrieval rather than autonomous interaction, TinyFish Fetch may be more appropriate. Its published standard limits are 150 URLs per minute and 1,000 URLs per day.

When the math shifts: Direct HTTP retrieval and optimized Playwright scripts may have a lower execution cost for predictable websites. Managed services become relevant when session management, browser hosting, authentication, or ongoing maintenance consumes substantial engineering time. Evaluate the total cost per successful result rather than assuming one method is always cheaper.

Handling the Mixed List

Real URL lists are rarely uniform. A supplier monitoring list might include:

• 60% static pricing pages (requests would work)
• 30% JavaScript-rendered product tables (Playwright needed)
• 10% authenticated portals with strict automation requirements (agents needed)

The practical approach is to classify URLs before processing the full list. Use a representative sample to determine whether the required content is present in the initial HTTP response, needs JavaScript rendering, or requires an authorized session or interactive workflow. A HEAD request alone does not establish which rendering method is necessary.

Route each category to the simplest suitable retrieval method. Reserve browser or agent execution for tasks that cannot be completed th

rough direct retrieval.

To classify URLs before routing them, a quick probe is faster than a full crawl:

A 429 response means rate-limited — retry with backoff before escalating. A 403 indicates access is blocked or restricted; retrying with the same tool won't help. A near-empty response or JS framework marker means JS rendering is needed. Clean HTML with visible tags is static.

Scale Considerations

For very large lists (100K+), distributed architecture matters regardless of tool—whether that's Scrapy's built-in scheduler, a task queue like Celery, or submitting batches to an async agent API and polling for results.

Large URL lists also benefit from deduplication, checkpointing, and partial-failure recovery. Save progress periodically so interrupted jobs can resume without repeating every request. Keep unsuccessful URLs and their errors separate from successful records to make retries easier to manage.

Try TinyFish Free

$8 allowance available, no credit card. The fastest way to test whether TinyFish fits your workflow.

Start free →

Related Reading

• The Best Web Scraping Tools in 2026
• How to Monitor 1,000 Websites in Parallel with the TinyFish API
• Scraping Dynamic Websites: When Playwright Is the Right Tool
• Why Your Stealth Plugin Isn't Working (And What Actually Does)