Debug Issues¶

Learn how to troubleshoot common problems and enable debug logging.

Enabling Debug Logging¶

Use debug=True to enable verbose logging:

from fastreq import fastreq

results = fastreq(
    urls=["https://httpbin.org/get"] * 5,
    debug=True,  # Enable debug logging
)

Debug output includes: - Backend selection - Request start/stop times - Retry attempts - Rate limiting info - Proxy usage - Error details

Disabling Progress Bars¶

Use verbose=False to disable progress bars:

from fastreq import fastreq

results = fastreq(
    urls=["https://httpbin.org/get"] * 100,
    verbose=False,  # Quiet mode, no progress bar
)

Checking Backend Availability¶

Verify which backends are installed:

from fastreq.backends import get_available_backends

available = get_available_backends()
print(f"Available backends: {available}")

# Expected output: ['niquests', 'aiohttp', 'requests']

Testing Backend Connectivity¶

Test if a backend works:

from fastreq import fastreq

def test_backend(backend_name):
    try:
        results = fastreq(
            urls=["https://httpbin.org/get"],
            backend=backend_name,
            timeout=5,
            debug=True,
        )
        print(f"{backend_name}: OK")
        return True
    except Exception as e:
        print(f"{backend_name}: FAILED - {e}")
        return False

for backend in ["niquests", "aiohttp", "requests"]:
    test_backend(backend)

Debugging Rate Limiting¶

See rate limiting behavior:

from fastreq import fastreq

results = fastreq(
    urls=["https://api.example.com/data"] * 20,
    rate_limit=5,
    rate_limit_burst=10,
    debug=True,  # Shows rate limiting logs
)

Example output:

[DEBUG] Rate limit: 5.0 req/s, burst: 10
[DEBUG] Request 1: immediate (bucket: 9/10)
[DEBUG] Request 2: immediate (bucket: 8/10)
...
[DEBUG] Request 11: waiting 0.2s for token

Debugging Retry Attempts¶

Monitor retry behavior:

from fastreq import fastreq

results = fastreq(
    urls=["https://api.example.com/unstable"],
    max_retries=3,
    debug=True,
)

Example output:

[DEBUG] Request 1: failed with 500, retry 1/3
[DEBUG] Request 1: failed with 500, retry 2/3
[DEBUG] Request 1: failed with 500, retry 3/3
[DEBUG] Request 1: exhausted retries

Debugging Proxy Issues¶

See which proxy is being used:

from fastreq import fastreq

results = fastreq(
    urls=["https://httpbin.org/ip"] * 5,
    proxies=[
        "http://proxy1:8080",
        "http://proxy2:8080",
    ],
    debug=True,
)

Common Issues and Solutions¶

Issue: Backend Not Available¶

Error: BackendError: No suitable backend found

Solution: Install backend dependencies:

pip install fastreq[all]

Or install specific backend:

pip install fastreq[niquests]
pip install fastreq[aiohttp]
pip install fastreq[requests]

Issue: Requests Timing Out¶

Error: TimeoutError or ReadTimeout

Solution: Increase timeout or check network:

from fastreq import fastreq

# Increase timeout
results = fastreq(
    urls=["https://httpbin.org/delay/10"],
    timeout=15,  # Increase from default
)

# Test network connectivity
import requests
requests.get("https://httpbin.org/get", timeout=5)

Issue: Rate Limit Exceeded¶

Error: RateLimitExceededError or HTTP 429

Solution: Adjust rate limiting:

from fastreq import fastreq

# Lower rate limit
results = fastreq(
    urls=["https://api.example.com/data"] * 100,
    rate_limit=5,           # Reduce from 10
    rate_limit_burst=5,      # Reduce from 10
    dont_retry_on=[429],     # Don't retry rate limit
)

Issue: Proxy Connection Failed¶

Error: ProxyError or ConnectionError

Solution: Test proxy configuration:

import requests

# Test proxy manually
try:
    response = requests.get(
        "https://httpbin.org/ip",
        proxies={"http": "http://proxy1:8080"},
        timeout=10,
    )
    print(f"Proxy working: {response.json()['origin']}")
except Exception as e:
    print(f"Proxy failed: {e}")

Issue: HTTP/2 Not Working¶

Error: HTTP/2 not available

Solution: Ensure niquests is installed:

# Install niquests
pip install fastreq[niquests]

# Verify
python -c "import niquests; print('niquests installed')"

from fastreq import fastreq

# Force niquests backend
results = fastreq(
    urls=["https://httpbin.org/get"],
    backend="niquests",
    debug=True,
)

Issue: SSL Certificate Errors¶

Error: SSLError or certificate validation failed

Solution: Verify SSL certificates or disable (not recommended for production):

from fastreq import fastreq

# Note: Disabling SSL verification is not recommended for production
# This is for debugging only
import ssl
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

Issue: Partial Failures¶

Error: PartialFailureError

Solution: Handle partial failures gracefully:

from fastreq import fastreq, PartialFailureError

urls = [
    "https://api.github.com/repos/python/cpython",
    "https://invalid-url.com",
    "https://api.github.com/repos/python/pypy",
]

try:
    results = fastreq(urls=urls)
except PartialFailureError as e:
    print(f"Partial failure: {e.successes}/{e.total}")
    print(f"Failed URLs: {e.get_failed_urls()}")

    # Optionally retry failed URLs
    failed_urls = list(e.get_failed_urls())
    if failed_urls:
        print(f"Retrying {len(failed_urls)} failed URLs...")
        results = fastreq(urls=failed_urls)

Issue: High Memory Usage¶

Error: Out of memory or slow performance

Solution: Use streaming for large files:

from fastreq import fastreq

# Use streaming instead of loading entire response
def stream_handler(response, url):
    with open(f"output_{url.split('/')[-1]}", "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            if chunk:
                f.write(chunk)

results = fastreq(
    urls=["https://example.com/large-file.zip"],
    return_type="stream",
    stream_callback=stream_handler,
)

Debugging with Context Manager¶

Use context manager for better debugging:

import asyncio
from fastreq import FastRequests

async def debug_with_context():
    async with FastRequests(debug=True) as client:
        # First batch
        results1 = await client.request(
            urls=["https://api.github.com/repos/python/cpython"],
        )

        # Second batch (reuses session)
        results2 = await client.request(
            urls=["https://api.github.com/repos/python/pypy"],
        )

        return results1, results2

results1, results2 = asyncio.run(debug_with_context())

Logging Configuration¶

Use Python logging for custom logging:

import logging
from fastreq import fastreq

# Configure logging
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Make requests
results = fastreq(
    urls=["https://httpbin.org/get"] * 5,
    debug=True,
)

Debug Checklist¶

Use this checklist when debugging:

Backend Available?

from fastreq.backends import get_available_backends
print(get_available_backends())

Network Working?

import requests
requests.get("https://httpbin.org/get", timeout=5)

URLs Valid?

from urllib.parse import urlparse
for url in urls:
    parsed = urlparse(url)
    print(f"{url}: {parsed.scheme}, {parsed.netloc}")

Proxy Working?

import requests
requests.get("https://httpbin.org/ip", proxies={"http": "http://proxy:8080"})

Rate Limits Correct?

# Check API documentation for rate limits
# Use rate_limit below documented limit

Best Practices¶

Enable Debug Logging Early: Turn on debug when troubleshooting
```
debug=True
```
Test Simple Cases First: Start with a single URL
```
fastreq(urls=["https://httpbin.org/get"])
```
Check Dependencies: Verify all backends are installed
```
pip install fastreq[all]
```
Monitor Memory: Use streaming for large responses
```
return_type="stream"
```

Handle Errors Gracefully: Use try-except blocks

except Exception as e:
    print(f"Error: {e}")