fix(link): address 4 codex [P2] findings on the rate limiter
All four were real bypass or fail-open paths on an endpoint about to be publicly exposed: - X-Forwarded-For was trusted from ANY peer. The origin also listens on the tailnet, so anyone reaching it directly could pick -- and rotate -- their own rate-limit key by sending a header. Now honored only when the socket peer is in a configured trusted_proxies list, and the last hop must parse as a real IP. trust_forwarded_for without trusted_proxies REFUSES startup. - Capacity eviction was fail-open and exploitable: an attacker able to mint many distinct keys could evict their own live window and start fresh. Now reclaims only EXPIRED windows and refuses the new key when all are live. Fail closed -- /v1/link is invite-only, so hitting the cap is an attack. - The clock was read outside the lock, so racing threads could append out of order; both retry_after (hits[0]) and reclamation (v[-1]) assume the list is chronological. Moved inside. - Config types were unvalidated: `"enabled": null` or `0` silently disabled limiting, and the string "false" enabled XFF trust (non-empty strings are truthy). Booleans must now be real JSON booleans; rate_limit must be an object. Codex confirmed no path-variant bypass (dispatch is exact-match) and no keep-alive/pipelining bypass (rejects set close_connection). Tests 87 -> 99: capacity fail-closed with the victim's window proven untouched through a 40-key flood, 200-thread chronological-order check, untrusted-peer spoof, junk XFF, and every config-type trap. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01LTARYHX7GPepi3CH3tp5pg
This commit is contained in:
Nirav Patel
co-authored by
Claude Opus 5
Claude Opus 5
parent
e2fed5886f
commit
c442721aff
+95
-33
@@ -52,6 +52,7 @@ Stdlib only. Python 3.9+.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import ipaddress
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
@@ -120,52 +121,93 @@ class RateLimiter:
|
||||
limit, window = rule
|
||||
if limit <= 0: # 0 = endpoint disabled entirely
|
||||
return False, window
|
||||
now = self._clock()
|
||||
key = (route, client)
|
||||
with self._lock:
|
||||
# Clock read INSIDE the lock: taken outside, two racing threads
|
||||
# can append out of order, and both hits[0] (retry_after) and
|
||||
# v[-1] (eviction age) assume the list is chronological.
|
||||
now = self._clock()
|
||||
hits = [t for t in self._hits.get(key, ()) if now - t < window]
|
||||
if len(hits) >= limit:
|
||||
self._hits[key] = hits
|
||||
return False, max(1, int(window - (now - hits[0])) + 1)
|
||||
if key not in self._hits and len(self._hits) >= self.max_keys:
|
||||
# At capacity, reclaim expired keys first...
|
||||
self._reclaim_expired(now)
|
||||
if len(self._hits) >= self.max_keys:
|
||||
# ...and if every window is still live, this is an
|
||||
# identity flood, not organic load. Evicting here would
|
||||
# let an attacker reset their own limit on demand, so
|
||||
# refuse the NEW key instead. Fail closed: /v1/link is
|
||||
# invite-only and low-volume, so hitting this cap is an
|
||||
# attack, and turning tokens away beats minting them.
|
||||
LOG.error("rate limiter at capacity (%d keys) with no "
|
||||
"expired windows: refusing new client %r on %s",
|
||||
self.max_keys, client, route)
|
||||
return False, window
|
||||
hits.append(now)
|
||||
self._hits[key] = hits
|
||||
if len(self._hits) > self.max_keys:
|
||||
self._prune(now)
|
||||
return True, 0
|
||||
|
||||
def _prune(self, now):
|
||||
"""Caller holds the lock. Drop expired keys; if still over the cap,
|
||||
drop the least-recently-hit. Dropping RESETS those clients' windows,
|
||||
which is fail-open -- but an unbounded dict is a memory DoS, and a
|
||||
bounded one is the lesser failure. Say so loudly either way."""
|
||||
for k in [k for k, v in self._hits.items()
|
||||
if not v or now - v[-1] >= self.rules[k[0]][1]]:
|
||||
def _reclaim_expired(self, now):
|
||||
"""Caller holds the lock. Drop only keys whose window has fully
|
||||
expired -- never a live one, or eviction becomes the bypass."""
|
||||
dead = [k for k, v in self._hits.items()
|
||||
if not v or now - v[-1] >= self.rules[k[0]][1]]
|
||||
for k in dead:
|
||||
del self._hits[k]
|
||||
if len(self._hits) > self.max_keys:
|
||||
over = len(self._hits) - self.max_keys
|
||||
for k in sorted(self._hits, key=lambda k: self._hits[k][-1])[:over]:
|
||||
del self._hits[k]
|
||||
LOG.warning("rate limiter at capacity (%d keys): dropped %d "
|
||||
"least-recent windows; those clients start fresh",
|
||||
self.max_keys, over)
|
||||
if dead:
|
||||
LOG.info("rate limiter reclaimed %d expired windows", len(dead))
|
||||
|
||||
|
||||
def client_ip(handler, trust_forwarded_for):
|
||||
def client_ip(handler, trust_forwarded_for, trusted_proxies=()):
|
||||
"""The address to rate-limit on.
|
||||
|
||||
Behind cloudflared every request arrives from the tunnel, so limiting on
|
||||
the socket peer would let one abuser starve everyone. X-Forwarded-For is
|
||||
client-controlled, though: a caller can prepend anything. A trusted proxy
|
||||
APPENDS the peer it actually saw, so the LAST entry is the only one we did
|
||||
not let the client choose -- take that, never the first. Off by default;
|
||||
only enable when something trustworthy really is in front.
|
||||
client-controlled, though: a caller can prepend anything. Two guards:
|
||||
|
||||
1. The header is honored ONLY when the socket peer is itself a configured
|
||||
trusted proxy. Without that, anyone who can reach the origin directly
|
||||
-- and the origin also listens on the tailnet -- picks their own
|
||||
rate-limit key and rotates it at will.
|
||||
2. A trusted proxy APPENDS the peer it actually saw, so the LAST entry is
|
||||
the only one the client did not choose. Take that, never the first.
|
||||
|
||||
The value must parse as a real IP; junk falls back to the socket peer
|
||||
rather than becoming a key of its own.
|
||||
"""
|
||||
if trust_forwarded_for:
|
||||
xff = handler.headers.get("X-Forwarded-For", "")
|
||||
parts = [p.strip() for p in xff.split(",") if p.strip()]
|
||||
if parts:
|
||||
return parts[-1]
|
||||
return handler.client_address[0]
|
||||
peer = handler.client_address[0]
|
||||
if not trust_forwarded_for:
|
||||
return peer
|
||||
if not _ip_in_any(peer, trusted_proxies):
|
||||
LOG.warning("X-Forwarded-For ignored: peer %s is not a trusted proxy",
|
||||
peer)
|
||||
return peer
|
||||
xff = handler.headers.get("X-Forwarded-For", "")
|
||||
parts = [p.strip() for p in xff.split(",") if p.strip()]
|
||||
if not parts:
|
||||
return peer
|
||||
try:
|
||||
return str(ipaddress.ip_address(parts[-1]))
|
||||
except ValueError:
|
||||
LOG.warning("X-Forwarded-For last hop %r is not an IP; using peer",
|
||||
parts[-1][:60])
|
||||
return peer
|
||||
|
||||
|
||||
def _ip_in_any(addr, networks):
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr)
|
||||
except ValueError:
|
||||
return False
|
||||
for net in networks:
|
||||
try:
|
||||
if ip in ipaddress.ip_network(net, strict=False):
|
||||
return True
|
||||
except ValueError:
|
||||
LOG.error("ignoring malformed trusted_proxies entry %r", net)
|
||||
return False
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -294,8 +336,26 @@ class LinkService:
|
||||
# Rate limiting is ON by default: this endpoint creates accounts and
|
||||
# mints tokens, so the safe default is limited, and disabling it has
|
||||
# to be a deliberate config act rather than an omission.
|
||||
rl = config.get("rate_limit") or {}
|
||||
self.trust_forwarded_for = bool(rl.get("trust_forwarded_for", False))
|
||||
rl = config.get("rate_limit", {})
|
||||
if rl is None:
|
||||
rl = {}
|
||||
if not isinstance(rl, dict):
|
||||
raise SystemExit("config rate_limit must be an object")
|
||||
# Real JSON booleans only. `"enabled": null` or `0` must not quietly
|
||||
# turn limiting off, and the string "false" must not turn XFF trust
|
||||
# ON (every non-empty string is truthy).
|
||||
for flag, default in (("enabled", True), ("trust_forwarded_for", False)):
|
||||
if flag in rl and not isinstance(rl[flag], bool):
|
||||
raise SystemExit(
|
||||
f"config rate_limit.{flag} must be true or false, "
|
||||
f"got {rl[flag]!r}")
|
||||
self.trust_forwarded_for = rl.get("trust_forwarded_for", False)
|
||||
self.trusted_proxies = tuple(rl.get("trusted_proxies") or ())
|
||||
if self.trust_forwarded_for and not self.trusted_proxies:
|
||||
raise SystemExit(
|
||||
"config rate_limit.trust_forwarded_for requires a non-empty "
|
||||
"trusted_proxies list -- trusting the header from any peer "
|
||||
"lets callers choose their own rate-limit key")
|
||||
if rl.get("enabled", True):
|
||||
rules = dict(DEFAULT_RATE_RULES)
|
||||
for route, spec in (rl.get("rules") or {}).items():
|
||||
@@ -308,9 +368,10 @@ class LinkService:
|
||||
f"[max_requests, window_seconds] with window > 0")
|
||||
rules[route] = (spec[0], spec[1])
|
||||
self.limiter = RateLimiter(rules)
|
||||
LOG.info("rate limiting active: %s (trust_forwarded_for=%s)",
|
||||
LOG.info("rate limiting active: %s (trust_forwarded_for=%s, "
|
||||
"trusted_proxies=%s)",
|
||||
{k: f"{v[0]}/{v[1]}s" for k, v in rules.items()},
|
||||
self.trust_forwarded_for)
|
||||
self.trust_forwarded_for, list(self.trusted_proxies))
|
||||
else:
|
||||
self.limiter = None
|
||||
LOG.warning("rate limiting DISABLED by config -- /v1/link will "
|
||||
@@ -571,7 +632,8 @@ class Handler(BaseHTTPRequestHandler):
|
||||
# to drain a body we already decided to reject.
|
||||
limiter = getattr(self.service, "limiter", None)
|
||||
if limiter is not None:
|
||||
who = client_ip(self, self.service.trust_forwarded_for)
|
||||
who = client_ip(self, self.service.trust_forwarded_for,
|
||||
self.service.trusted_proxies)
|
||||
allowed, retry_after = limiter.check(self.path, who)
|
||||
if not allowed:
|
||||
LOG.warning("rate limited %s %s (retry after %ss)",
|
||||
|
||||
Reference in New Issue
Block a user