feat: Add opt-in per-domain request throttling for HTTP 429 backoff#1762

Open

MrAliHasan wants to merge 12 commits intoapify:masterfrom

MrAliHasan:fix/request-throttler-429-backoff

Commits on Feb 20, 2026

fix: add per-domain RequestThrottler for 429 backoff (#1437)
MrAliHasan
committed

Commits on Feb 23, 2026

refactor: replace RequestThrottler with ThrottlingRequestManager
MrAliHasan
committed

Commits on Feb 25, 2026

refactor: reimplement `ThrottlingRequestManager` with per-domain sub-queues and update its integration across crawlers.
MrAliHasan
committed

Commits on Feb 26, 2026

test: fix typing and linting checks in ThrottlingRequestManager tests
MrAliHasan
committed

Commits on Mar 2, 2026

feat: Add explicit domain routing and management to the request throttling manager.
MrAliHasan
committed

Commits on Mar 4, 2026

feat: Implement recreate_purged for ThrottlingRequestManager and refactor domain state management and sub-queue handling.
MrAliHasan
committed

Commits on Mar 5, 2026

deps: Pin ty to version 0.0.18 and update uv.lock to include Python 3.14 wheels for brotlicffi.
MrAliHasan
committed

Commits on Mar 6, 2026

fix: Ensure ThrottlingRequestManager.add_request explicitly handles None returns and updates the ty dependency constraint.
MrAliHasan
committed

Commits on Mar 11, 2026

refactor: Address reviewer feedback on ThrottlingRequestManager and revert uv.lock
MrAliHasan
committed

Commits on Mar 13, 2026