Make Parallel Requests¶

Learn how to create parallel HTTP requests with different configurations and return types.

Basic Parallel Requests¶

Make multiple requests to the same or different URLs:

from fastreq import fastreq

# Single URL repeated
results = fastreq(
    urls=["https://httpbin.org/get"] * 5,
    concurrency=3,
)

# Multiple different URLs
results = fastreq(
    urls=[
        "https://api.github.com/repos/python/cpython",
        "https://api.github.com/repos/python/pypy",
        "https://api.github.com/repos/legout/fastreq",
    ],
)

Named Results with Keys¶

Use the keys parameter to return a dictionary mapping names to results:

from fastreq import fastreq

results = fastreq(
    urls=[
        "https://api.github.com/repos/python/cpython",
        "https://api.github.com/repos/python/pypy",
    ],
    keys=["cpython", "pypy"],
)

# Access results by key
print(f"CPython: {results['cpython']['name']}")
print(f"PyPy: {results['pypy']['name']}")

The keys parameter is useful when you need to track which result corresponds to which request.

Custom Response Transformation¶

Use parse_func to apply custom transformation to each response:

from fastreq import fastreq

def extract_repo_info(response):
    """Extract only relevant fields from GitHub API response."""
    return {
        "name": response.get("name"),
        "stars": response.get("stargazers_count"),
        "language": response.get("language"),
    }

results = fastreq(
    urls=[
        "https://api.github.com/repos/python/cpython",
        "https://api.github.com/repos/legout/fastreq",
    ],
    parse_func=extract_repo_info,
)

for repo in results:
    print(f"{repo['name']}: {repo['stars']} stars, {repo['language']}")

Different Return Types¶

Control how responses are parsed with return_type:

JSON Response (Default)¶

from fastreq import fastreq

results = fastreq(
    urls=["https://api.github.com/repos/python/cpython"],
    return_type="json",
)

print(results[0]["name"])  # Access JSON fields directly

Text Response¶

results = fastreq(
    urls=["https://httpbin.org/html"],
    return_type="text",
)

print(results[0])  # Raw HTML text

Binary Content¶

results = fastreq(
    urls=["https://httpbin.org/bytes/1024"],
    return_type="content",
)

print(len(results[0]))  # Length in bytes

Full Response Object¶

from fastreq import fastreq, ReturnType

results = fastreq(
    urls=["https://httpbin.org/get"],
    return_type=ReturnType.RESPONSE,
)

response = results[0]
print(f"Status: {response.status_code}")
print(f"Headers: {response.headers}")
print(f"Body: {response.json()}")

Async Parallel Requests¶

For async applications, use the async version:

import asyncio
from fastreq import fastreq_async

async def fetch_data():
    results = await fastreq_async(
        urls=[
            "https://api.github.com/repos/python/cpython",
            "https://api.github.com/repos/python/pypy",
        ],
        concurrency=5,
    )
    return results

results = asyncio.run(fetch_data())

Using a Context Manager¶

Reuse sessions across multiple request batches:

import asyncio
from fastreq import FastRequests

async def fetch_with_session():
    async with FastRequests(concurrency=5) as client:
        # First batch
        results1 = await client.request(
            urls=["https://api.github.com/repos/python/cpython"],
        )

        # Second batch (reuses session/cookies)
        results2 = await client.request(
            urls=["https://api.github.com/repos/python/pypy"],
        )

        return results1, results2

results1, results2 = asyncio.run(fetch_with_session())

Mixing URL Parameters¶

Pass URL-specific parameters using dictionaries:

from fastreq import fastreq

results = fastreq(
    urls=[
        "https://api.github.com/repos/python/cpython",
        "https://api.github.com/repos/python/pypy",
    ],
    params=[
        {"ref": "main"},  # First URL params
        {},               # Second URL params (empty)
    ],
)

Combining with Other Options¶

Combine parallel requests with other features:

from fastreq import fastreq

results = fastreq(
    urls=[
        "https://api.github.com/repos/python/cpython",
        "https://api.github.com/repos/python/pypy",
    ],
    keys=["cpython", "pypy"],
    concurrency=3,
    max_retries=2,
    timeout=10,
    rate_limit=10,
    headers={
        "User-Agent": "MyApp/1.0",
        "Accept": "application/vnd.github.v3+json",
    },
)