Module 1: Production Python

Read the transcript

1. Why ‘Python is easy’ isn’t the same as ‘Python in production is easy’

Host: Python won the AI infrastructure war, basically by default. Every framework worth using has a Python API first, and the ecosystem gives you a direct line to every ML tool that matters. But there’s a dangerous assumption baked into that popularity: that because Python is easy to write, it’s easy to run in production, especially when you’re holding a thousand concurrent connections to model providers that are slow and occasionally just fall over.

Guest: Right, and that gap is exactly what this episode is about. Knowing when async actually helps versus when it’s the wrong tool, what the GIL really constrains you to do, and how to build something that degrades predictably instead of quietly collapsing under load — none of that comes for free just because the language is friendly. And we’re not doing this with toy snippets today. Everything we walk through comes straight out of a real lab, which you can find under the labs folder, named async dash a i dash gateway, with its own test suite and CI, so when we show a pattern, it’s because it’s been proven to actually work, not just because it looks clean on a slide.

2. Choosing a concurrency model by workload, not fashion

Host: Okay, so before we look at any gateway code, let’s settle the concurrency question, because I think a lot of people just default to asyncio because it’s the trendy choice. Walk me through how you’d actually pick between asyncio, threads, processes, and external workers.

Guest: You pick based on the workload, not the hype. Asyncio wins when you’ve got high-concurrency I/O — calling model APIs, hitting a database, streaming responses — but the catch is it’s cooperative, so one blocking call in a shared loop stalls every other coroutine waiting on it. Threads are your fallback for blocking libraries that have no async API, but you’re now managing shared-memory risk, and the GIL means you get zero throughput gain if that thread work is pure-Python CPU crunching. Processes give you real parallelism for CPU-bound Python and isolation, at the cost of serialization overhead and slow startup, and external workers are for durable, retryable, independently scalable jobs like ingestion or batch inference — wrong tool entirely if you need a synchronous request/response answer right now.

Host: So let’s demystify the GIL itself, because I think people hear ‘global interpreter lock’ and just assume Python can’t do anything at once.

Guest: That’s the myth worth killing. The GIL means only one thread executes Python bytecode at a time within a process — but it doesn’t block I/O concurrency, because while one coroutine is awaiting a network response, others run freely, and it doesn’t block parallelism in native-code extensions like NumPy or most ML runtimes, since they release the GIL during their C or C++ work. It’s a constraint specifically on pure-Python CPU-bound parallelism, not a blanket ‘Python is single-threaded’ statement. Which is why the one sentence that should drive every design choice here is: the event loop is a scheduler, not a speed multiplier — a coroutine runs until it hits an await, hands control back, and concurrency comes purely from overlapping wait time, not from CPU-heavy code you accidentally stuffed into an async handler.

3. Inside the gateway: the request’s actual admission-and-execution path

Host: So let’s actually walk the path a request takes through generate() — not the theory, the real branches in the code. What’s the first gate it hits?

Guest: Rate limiting, before anything else touches concurrency admission. That ordering is deliberate: if a request is going to get rejected for quota reasons, it should never occupy a semaphore slot first, because that’s capacity a legitimate request needed and now can’t get. Check quota, reject fast if you’re over, and only then try to acquire a concurrency slot.

Host: Okay, so it survives rate limiting and grabs a slot — then it’s into the retry loop under a deadline. Where do people usually get that wrong?

Guest: They wrap the timeout around each individual attempt instead of the whole loop. The async timeout context has to span every retry, because a caller’s fifteen-second budget is for the entire operation — three retries at fifteen seconds each silently becomes forty-five seconds nobody asked for. And no matter how that loop exits — success, timeout, exception, cancellation — the semaphore release lives in a finally block, so capacity always comes back instead of leaking away request by request.

4. Coroutines, layered timeouts, cancellation, and backpressure

Host: Let’s back up to something you slipped in earlier — coroutines versus tasks. When I write an async def and call it, is that thing actually running yet?

Guest: No, and that trips people up constantly. Calling an async function just returns a coroutine object — it’s inert until something awaits it or schedules it as a Task. A coroutine floating around with no owner and no error handling is a bug waiting to happen, because nothing is watching it or propagating its failure.

Host: So is that the case for structured concurrency over just throwing things at gather()?

Guest: Exactly. gather() will happily keep awaiting the remaining tasks even after one of them blows up, so you get a silent straggler. TaskGroup ties tasks to a lifecycle scope — first failure cancels its siblings predictably, and that cancellation isn’t an error state, it’s just CancelledError propagating through every await point, which your finally blocks should clean up after and then re-raise, never swallow.

5. Reading the real code: _admit, generate, and the circuit breaker

Host: Let’s ground this in the actual code, because _admit is doing something a lot of people get backwards. Walk me through the order of operations there.

Guest: We actually walked through that ordering and the wait_for timeout behavior in the earlier segment on the gateway’s admission-and-execution path, so I won’t retread it here — same logic applies in _admit.

Host: And then inside generate, there’s this branch where a pinned provider_name skips the whole retry loop. That feels like it’d be tempting to ‘fix’ by just retrying anyway.

Guest: Tempting and wrong. If a caller explicitly asked for provider X, silently retrying into a fallback and returning a response from provider Y is a correctness bug wearing a resilience costume — they’d get an answer, just not the one they thought they were getting. So a pinned request gets one shot and an honest, fast failure. Everyone else gets the jittered backoff, base times two to the attempt minus one, randomized so concurrent retries don’t all slam the provider on the same beat — and that whole per-request loop lives inside asyncio.timeout, so retries can’t quietly outlive the deadline.

Host: That backoff protects one provider from one client’s retries. What stops every client’s retries from hammering a provider that’s already down?

Guest: That’s the circuit breaker, and the clever bit is the state property is computed, not stored. When it’s OPEN it checks time.monotonic() minus when it opened against the recovery timeout, and if enough time’s passed it just reports HALF_OPEN on that read — no background task, no timer thread, nothing to leak or forget to cancel. HALF_OPEN then lets exactly one probe through, gated by a lock, so you test whether the provider’s healthy again without every waiting caller piling onto it the instant the window opens.

6. Draining without dropping requests on deploy

Host: So the circuit breaker handles a misbehaving provider, but what handles a misbehaving deploy? Every rolling update sends SIGTERM eventually, and I’ve seen plenty of services just eat in-flight requests when that happens.

Guest: Right, that’s what draining.py is for, and the pattern is almost embarrassingly simple once you see it. Every request wraps itself in an async context manager that increments an active counter going in and decrements it in a finally block coming out, so success or failure, it always deregisters. Then drain flips an accepting flag to false first, so the load balancer stops sending new traffic, and only after that does it wait for the active count to reach zero.

Host: And that wait isn’t open-ended, presumably — you said bounded a second ago.

Guest: Right, it’s wrapped in asyncio.timeout with a default of thirty seconds, waiting on a condition variable for active to hit zero. If everything drains in time it returns True; if the timeout fires it returns False, and that boolean is the whole point — instead of the process just getting SIGKILLed with no record, you get an explicit, checkable signal that some requests were cut off, which you can log, alert on, or feed into your deploy tooling.

7. How these systems actually fail

Host: Okay, so we’ve built all this careful machinery — timeouts, draining, the breaker. Where does it actually break in practice? What’s the first failure mode you see in real deployments?

Guest: The classic one is a blocking call hiding inside an async function — a synchronous HTTP client, a file read, some CPU-heavy parsing loop. That doesn’t just slow down its own request, it freezes the entire event loop, so every other coroutine sharing that loop stalls too. The fix is boring but non-negotiable: blocking work goes through run_in_executor or a separate process, never inline.

Host: And that semaphore we talked about for admission control — I assume people find ways around it without meaning to?

Guest: Constantly. Someone calls asyncio dot gather over a big unbounded list of coroutines, and if those tasks aren’t each individually going through _admit, you’ve just bypassed the semaphore entirely and blown through your connection limits and provider quotas in one shot. Same family of bug as a leaked semaphore acquire with no release on some exit path — works fine in testing, then hours later under real load you get connection exhaustion and a shutdown that behaves nothing like you expect. And separately, synchronized retries without jitter turn a blip into an outage, which is why gateway.py uses random.uniform backoff and trips the circuit breaker instead of just hammering a struggling provider harder — and why your dashboard can lie to you: p50 looks fine while p99 and queue-wait time are quietly climbing, so you have to track those, and saturation and cancellations, as first-class numbers, not averages.

8. Trade-offs: there is no universally correct default

Host: Let’s talk about the rate limiter, because I noticed TenantRateLimiter in rate_limit.py is just an in-process token bucket. That feels like it’d fall apart the second you run more than one replica.

Guest: It does, and that’s the point worth internalizing: it’s correct and fast for exactly the lab’s single-process setup, and wrong the moment you scale out, because each replica enforces the quota independently — a tenant effectively gets a multiple of their real limit proportional to the number of replicas running. The lab actually includes the fix too, a Redis-backed limiter that does an atomic refill-and-consume in one Lua script, but that trades the correctness problem for a new dependency and a new failure mode: what does enforcement even mean when Redis itself is unavailable. Neither one is strictly better, the in-process version is just the right default until you actually have more than one replica, and you swap it out when reality demands it, not before.

Host: That same ‘no universal default’ logic seems to apply to threads versus async versus processes too, and to how much type-safety ceremony you bother with — where do you draw those lines?

Guest: Same instinct exactly. Typing follows that same pragmatism — Pydantic models and full annotations at every service boundary, request and response models, provider interfaces, because that’s where bugs slip through and where a stranger needs to understand the contract without reading the implementation, but inside a single function’s local logic you can loosen up, because that ceremony there is just friction with no payoff.

9. Security boundaries a gateway can’t skip

Host: Let’s talk security, because a gateway is basically a funnel for external input, and Python has a few loaded guns lying around that ‘quick script’ habits leave lying around too. Where do you even start — the code, or something more boring than that?

Guest: Boring first, always — dependency supply chain. A lockfile, uv.lock or poetry.lock, with a CI step that actually verifies it rather than just running pip install with no pins, because an unpinned transitive dependency is how someone else’s compromised package becomes your RCE. Then secrets: never in code or committed config, always injected at runtime from env vars or a secrets manager — the lab’s SECURITY_AND_RUNTIME doc shows exactly how the JWT secret is handled that way. From there it’s Pydantic models validating every request body at the boundary instead of manual dict parsing, so garbage gets rejected before it touches business logic; never pickle untrusted input or run eval or exec anywhere near user data, both are classic RCE traps that sneak in through ‘quick’ scripts; and lock down outbound HTTP so a caller can’t steer which upstream URL you call, directly or through a redirect, or your gateway becomes a free port scanner into your own internal network.

10. Profiling, pooling, uvloop, and scaling on the right signals

Host: Let’s talk speed, but I want to head off the instinct to just start tweaking code. Where does someone actually start when the gateway feels slow?

Guest: With a profiler, not a guess — py-spy for production because it samples without touching the running process, cProfile in dev, and asyncio’s own debug mode to catch coroutines blocking the loop longer than they should. And don’t measure one end-to-end latency number; break it into connect time, queue wait, provider execution, serialization, streaming duration, because ‘requests are slow’ hides which piece is actually the culprit. Two cheap wins once you know where to look: reuse httpx.AsyncClient instances instead of creating one per request since they pool connections internally, and drop in uvloop as your event loop — near-zero code change, meaningful throughput gain on I/O-heavy workloads.

Host: Okay, so that’s making one instance faster. When it’s time to run more instances, what’s the right way to scale this thing out?

Guest: Stateless workers first — the gateway’s designed to be replica-safe as long as you avoid in-process state like that rate limiter we mentioned earlier. And autoscale on saturation signals — queue depth, semaphore wait time, p99 latency — not CPU, because an I/O-bound async service can sit at flat CPU right up until it collapses. Keep CPU-bound work like embedding generation or batch inference on a separate multi-process worker fleet, not inside the request-handling event loop, and as concurrency climbs, watch the edges — thread pools, DB connection pools, file descriptor limits — because ‘just add more semaphore capacity’ eventually slams into one of those ceilings.

11. Interview-ready answers and a hands-on checklist

Host: Let’s do a quick lightning round, because these are the exact questions that come up when someone’s screening for this kind of role. First one: why does time.sleep in an async handler bring everything down, not just that one request?

Guest: That one we already covered, so let’s jump to the follow-ups, since they travel together: semaphore versus token bucket is ‘in-flight work’ versus ‘arrival rate’ — a semaphore of 32 with no rate limiter still queues a burst of a thousand requests instead of rejecting any; TaskGroup over gather when one failure should cancel its siblings, gather with return_exceptions only when you’re deliberately inspecting partial results; and shutdown follows that same sequence we already walked through, with an explicit signal that draining actually finished rather than a silent kill.

Host: That last one is basically the DrainManager we already read. So if someone wants to internalize all of this instead of just reciting it back in an interview, what’s the actual hands-on move?

Guest: Clone the lab and run the checklist against your own service, line by line: explicit timeouts on every remote call, concurrency bounded by measured capacity not a guess, retries with capped backoff and jitter respecting a deadline, every finally releasing what its try acquired, CancelledError re-raised never swallowed, p95/p99 and queue-wait and cancellation counts actually tracked, and shutdown draining within a bounded window. Then go further — add a queue-wait metric separate from total latency and write a test proving it’s reported even when the provider call itself is fast, because that’s the hidden-queueing distinction that separates ‘looks fine in the dashboard’ from ‘is actually fine.’

12. Inside the lab: running it, the CI job that tested nothing, and principal-level questions

Host: So let’s get concrete about the lab itself. There are two entry points in there, production_app and secure_app — walk me through why you’d split them instead of just shipping one gateway.

Guest: production_app has everything we’ve talked about — bounded concurrency, deadlines, retries with jitter, health-aware fallback, RED metrics. secure_app layers JWT-verified tenant identity, tier-based policy, and a Redis-backed distributed rate limiter on top of that. We kept them separate so the identity and quota layer is legible on its own, but if you’re deploying this for real, you collapse them into one app with security always on — secure_app is the one to read as the reference.

Host: You clone it, stand up the venv, run docker compose for the full stack with Redis and the collector, and then pytest, ruff, mypy. That’s the standard loop. But you told me there’s a story about a CI job that was green for the wrong reason — what happened there?

Guest: The Redis integration job spun up a service container and then ran pytest -k redis, but the only tests matching that filter were backed by a FakeRedis stub — canned answers, no socket ever opened. The container sat there untouched; the job was green whether or not Redis existed, including if it had never started. The fix was a real test that fires two limiter instances at one Redis, capacity times two acquires concurrently, and checks that exactly capacity succeeds — swap the atomic Lua script for a naive get-then-set and 40 of 40 get through instead of 20, so the test actually measures the atomicity claim instead of restating it. The quieter part of the fix is REDIS_INTEGRATION_REQUIRED — under that flag a missing REDIS_URL fails the job instead of skipping it, because a skip doesn’t turn a build red, and CI is set on every job including the broken one.

Host: That’s a good note to end on — a green checkmark isn’t proof of anything by itself. Last thing: if someone wants to go from this episode to sounding principal-level in an interview, what do they walk through?

Guest: We’ve already covered the five things that matter here — the concurrency-versus-rate-limiting split, the atomic quota mutation, the health-routing feedback loop, the streaming fallback problem, and readiness versus liveness during a rolling deploy. Clone the lab, read PRODUCTION_READINESS.md against it, and treat every unchecked box as a gap you now know how to name.

Not covered

The planner wanted these and found nothing in the source to support them:

A guest interview with the lab’s original author about design history
Live benchmarking of uvloop vs standard event loop performance numbers

Generated from this page by Claude Sonnet 5 on 2026-08-22, spoken by Kokoro-82M running locally. Two synthetic voices, not a recorded conversation. Every claim is drawn from this page — where it differs from the text above, the text is correct.

Executive Summary

Python dominates AI application engineering — the ecosystem, the readability, the direct line to every ML framework worth using. None of that removes the need to understand what actually happens when your gateway holds a thousand concurrent connections to slow, unreliable model providers. Production Python for AI infrastructure means knowing when async/await helps and when it’s the wrong tool, how the GIL actually constrains your architecture, and how to design a system that degrades predictably instead of falling over silently under load.

This module builds those mental models using real code, not toy snippets — every example below is drawn directly from labs/async-ai-gateway, a lab with its own test suite and CI, not sample code invented for this page.

Mental Model

Choose a concurrency model by workload profile, not by which one is fashionable this year:

Model	Best for	The catch
asyncio	High-concurrency I/O: model APIs, databases, queues, sockets, streaming	Cooperative scheduling — one blocking call stalls every other coroutine sharing the loop
Threads	Blocking libraries with no async API	Shared memory means synchronization risk; the GIL means no throughput gain on pure-Python CPU work
Processes	CPU-bound Python work, isolation, real parallelism across cores	Serialization, startup cost, and cross-process coordination overhead
External workers	Durable, retryable, independently scalable tasks — ingestion, embedding, batch inference	Adds a queue and a worker fleet to operate; wrong choice for synchronous request/response paths

What the GIL actually changes

The Global Interpreter Lock means only one thread executes Python bytecode at a time within one process. It does not prevent I/O concurrency — while one coroutine awaits a network response, others run freely — and it does not prevent parallelism in native-code extensions (NumPy, most ML runtimes) that release the GIL during their C/C++ work. The GIL is a constraint on pure-Python CPU-bound parallelism specifically, not a blanket statement about Python being single-threaded.

The one-sentence version that should drive every design choice in this module: the event loop is a scheduler, not a speed multiplier. A coroutine runs until it hits an await; control returns to the loop so another ready task can run. Concurrency comes from overlapping waiting time across many requests — it does nothing for CPU-bound work, which is why CPU-heavy code inside an async handler is one of this module’s central failure modes.

Architecture

Here’s the actual admission-and-execution path a request takes through AsyncAIGateway.generate() — every box below is a real control-flow branch in that method, not a simplification:

flowchart TD
    A[Request arrives] --> B{Rate limiter has tokens?}
    B -->|No| R1[Reject: rate limited]
    B -->|Yes| C{Semaphore has capacity within queue-wait timeout?}
    C -->|No| R2[Reject: gateway overloaded]
    C -->|Yes| D[asyncio.timeout deadline starts]
    D --> E[Select provider]
    E --> F[Await provider call]
    F -->|Success| G[Record success, release capacity]
    F -->|ConnectionError or TimeoutError| H{Attempts remaining and provider not pinned?}
    H -->|Yes| I[Jittered exponential backoff, retry]
    I --> E
    H -->|No| J[Record failure, release capacity]
    G --> K[Return response]
    J --> R3[Raise to caller]

Three architectural decisions are doing the load-bearing work here, and each maps to a concept the Deep Dive section covers in full:

Rate limiting happens before concurrency admission. A rejected request due to quota should never occupy a semaphore slot — that’s wasted capacity a legitimate request needed.
The deadline (asyncio.timeout) wraps the entire retry loop, not each individual attempt. A caller’s 15-second budget is a budget for the whole operation, including every retry — otherwise three retries at 15 seconds each could keep a request alive for 45 seconds no caller asked for.
Capacity is released in a finally block, so a raised exception, a timeout, or a cancellation all still return the semaphore slot. Forgetting this is exactly the “leaked resources” failure mode covered later.

Deep Dive

Coroutines, tasks, and task groups. A function declared async def returns a coroutine object when called — nothing runs yet. awaiting it, or scheduling it as a asyncio.Task, is what actually starts execution. A bare coroutine with no owner and no error handling is a bug waiting to happen; prefer structured concurrency (asyncio.TaskGroup, Python 3.11+) which ties a group of tasks to a lifecycle scope and propagates the first failure predictably, cancelling its siblings — instead of asyncio.gather()’s default behavior of silently continuing to await the others.

Timeouts, layered. A single “timeout” is rarely one number. Production systems separate:

Queue-wait timeout — how long a request waits for a concurrency slot before being rejected as overloaded (max_queue_wait_seconds in the gateway above).
Per-attempt timeout — how long one provider call gets before it’s considered failed.
Total request deadline — the budget for the whole operation, retries included.

Conflating these is how a “20 second timeout” quietly becomes a 90-second wait under retries, which is worse for users than a fast, clear failure.

Cancellation is normal control flow, not an error. When a task is cancelled — because its deadline expired, or its caller disconnected — asyncio.CancelledError propagates through every await point inside it. Cleanup belongs in finally blocks and async context managers, and a CancelledError should almost always be re-raised, never swallowed (see the Failure Modes section for what happens when it’s caught and ignored).

Backpressure. Bounded asyncio.Semaphores and bounded queues are what stop an overloaded downstream dependency from turning into unbounded memory growth in your own process — the same mechanism the Architecture section’s _admit() step relies on.

Research Note

The essay that popularized TaskGroup-style structured concurrency in Python. Worth reading even if you never touch Trio — the argument for why unstructured create_task() calls are the async equivalent of goto applies directly to asyncio.

Source: Nathaniel J. Smith, "Notes on structured concurrency, or: Go statement considered harmful"

Implementation

The bounded-concurrency, deadline, and retry logic from the Architecture diagram, as it actually exists in the lab — not a simplified rewrite:

Admission control and retry, from gateway.py

async def _admit(self) -> None:
    if self._closing:
        raise RuntimeError("Gateway is shutting down")
    if self._rate_limiter is not None:
        await self._rate_limiter.acquire()
    try:
        await asyncio.wait_for(
            self._capacity.acquire(), timeout=self._max_queue_wait_seconds
        )
    except TimeoutError as exc:
        raise GatewayOverloadedError("Gateway concurrency limit reached") from exc

async def generate(
    self, prompt: str, *, provider_name: str | None = None, timeout_seconds: float = 15.0,
) -> GenerateResponse:
    started = time.perf_counter()
    await self._admit()
    try:
        async with asyncio.timeout(timeout_seconds):
            last_error: Exception | None = None
            for attempt in range(1, self._max_attempts + 1):
                provider = self._select(provider_name)
                try:
                    result = await provider.generate(prompt)
                    return GenerateResponse(
                        provider=result.provider, text=result.text,
                        attempts=attempt, latency_ms=(time.perf_counter() - started) * 1000,
                    )
                except (ConnectionError, TimeoutError) as exc:
                    last_error = exc
                    if provider_name is not None or attempt == self._max_attempts:
                        raise
                    backoff = self._base_backoff_seconds * (2 ** (attempt - 1))
                    await asyncio.sleep(random.uniform(0, backoff))
            if last_error is not None:
                raise last_error
    finally:
        self._capacity.release()

Two details that separate this from a “works on my machine” version: rate limiting happens before the semaphore acquire (rejected requests don’t consume concurrency capacity), and an explicitly pinned provider_name skips retries entirely — a caller who asked for a specific provider gets a fast, honest failure instead of a silent fallback to a different model they didn’t ask for.

The companion resilience primitive is a circuit breaker, so a provider that’s already failing doesn’t get hammered by every retry from every concurrent request:

Circuit breaker states, from resilience.py

class CircuitState(StrEnum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@property
def state(self) -> CircuitState:
    if self._state is CircuitState.OPEN:
        if time.monotonic() - self._opened_at >= self.recovery_timeout_seconds:
            return CircuitState.HALF_OPEN
    return self._state

The state property computes HALF_OPEN on read rather than storing it — the circuit doesn’t need a timer or background task to “wake up,” it just checks elapsed time whenever anyone asks. One probe request is allowed through in HALF_OPEN (guarded by an asyncio.Lock, in the full implementation) to test recovery without letting a thundering herd of retries all hit the recovering provider at once.

Production Example

Graceful shutdown is where a lot of “it works in dev” Python services fall apart in production — a rolling deploy sends SIGTERM, and if in-flight requests aren’t drained, they get killed mid-response. Here’s the real pattern from draining.py:

@asynccontextmanager
async def request(self):
    async with self._condition:
        if not self._accepting:
            raise RuntimeError("Gateway is draining")
        self._active += 1
    try:
        yield
    finally:
        async with self._condition:
            self._active -= 1
            if self._active == 0:
                self._condition.notify_all()

async def drain(self, timeout_seconds: float = 30.0) -> bool:
    async with self._condition:
        self._accepting = False
        if self._active == 0:
            return True
        try:
            async with asyncio.timeout(timeout_seconds):
                await self._condition.wait_for(lambda: self._active == 0)
            return True
        except TimeoutError:
            return False

The shape: every request registers itself as “active” on entry and deregisters on exit — success or failure, via the finally block. drain() flips readiness to false (so a load balancer stops routing new traffic once it notices), then waits, bounded by a timeout, for the active count to hit zero. If the timeout fires first, drain() returns False — an explicit, checkable signal that some requests were forcibly cut off, rather than a silent process kill with no record of what was lost.

Failure Modes

Blocking the event loop

A synchronous HTTP client, blocking file I/O, or a CPU-heavy parsing loop called directly inside an async def function freezes every other coroutine sharing that event loop — not just the slow request. Blocking work belongs in a thread-pool executor (loop.run_in_executor) or a separate process, never inline.

Unbounded fan-out

asyncio.gather() over an unbounded list of coroutines can exhaust connections, provider quotas, memory, and downstream capacity in one call. The semaphore in the Architecture section exists specifically to prevent this — an unbounded gather() bypasses it entirely if the tasks aren’t each individually going through _admit().

Retry storms

Immediate, synchronized retries across many concurrent requests amplify an outage instead of recovering from it — this is exactly why gateway.py uses random.uniform(0, backoff) jitter rather than a fixed backoff, and why the circuit breaker exists at all: past a failure threshold, stop retrying and fail fast instead of adding more load to an already-struggling provider.

Leaked resources

Unclosed HTTP clients, streams, or sockets — or a semaphore acquire() without a matching release() on every exit path — cause connection exhaustion and unpredictable shutdown behavior under load, often hours after the code that leaked them shipped.

Hidden queueing

Latency can accumulate before a request even starts executing. If you only measure wall-clock latency, a system stuck queueing for concurrency slots looks identical to one that’s actually slow to process — measure queue-wait time separately from provider execution time, or you’ll optimize the wrong thing.

Average latency hides the failure

A p50 latency dashboard can look perfectly healthy while p99 latency — the requests actually timing out or getting retried — is climbing. Track p95/p99, concurrency saturation, and cancellation/timeout counts as first-class metrics, not just mean latency.

Trade-offs

In-process vs. distributed rate limiting

rate_limit.py’s TenantRateLimiter is an in-process token bucket — correct and fast for a single-process lab, but wrong for a multi-replica deployment: each replica enforces the quota independently, so a tenant effectively gets replica_count × their real limit. The lab’s own distributed limiter (Redis-backed, atomic refill-and-consume in one Lua script) fixes that at the cost of a new dependency and a new failure mode — what happens to enforcement when Redis itself is unavailable. Neither option is strictly better; the in-process version is the right default until you have more than one replica.

asyncio vs. threads vs. processes

Async wins for high-concurrency I/O-bound work with async-native libraries. Threads earn their keep only when a required library is blocking with no async equivalent — they don’t improve pure-Python CPU throughput because of the GIL. Processes are the right (and only) answer for CPU-bound Python work, at the cost of serialization and startup overhead per process.

Strict typing overhead vs. velocity

Pydantic models and full type annotations catch a class of bugs before they reach production and make a codebase navigable by someone who didn’t write it — at the cost of upfront ceremony that feels like friction on a throwaway script. The right line: typed at every service boundary (request/response models, provider interfaces), looser inside a single function’s local logic.

Security

Dependency supply chain. Lockfiles (uv.lock, poetry.lock, or pinned requirements.txt) and a CI step that actually checks them, not just a pip install with no version pins.
Secrets never in code or config committed to the repo. Environment variables or a secrets manager, injected at runtime — see the lab’s own SECURITY_AND_RUNTIME.md for the concrete JWT-secret handling this applies to.
Input validation at every boundary. Pydantic models on request bodies, not manual dict parsing — reject malformed input before it reaches business logic, not after.
Never deserialize untrusted input with pickle, and avoid eval/exec on anything that touches user input — both are common in “quick” Python scripts and both are remote-code-execution risks in a service that accepts external requests.
SSRF risk in outbound HTTP clients. A gateway that lets a caller influence which upstream URL it calls (directly or via a redirect it follows) can be turned into a proxy for internal network scanning — validate and allowlist upstream targets.

Performance

Profile before optimizing — py-spy for production-safe sampling profiling without code changes, cProfile for development, and asyncio’s own debug mode (PYTHONASYNCIODEBUG=1) to catch coroutines that block the loop for too long.
Decompose latency, don’t just measure it end-to-end — connect time, queue wait, provider execution time, serialization, and streaming duration are each worth their own metric; a single “request latency” number hides which one actually needs fixing.
Connection pooling — reuse httpx.AsyncClient instances (they pool connections internally) instead of creating a new client per request, which pays connection-setup cost on every call.
uvloop as a drop-in event loop replacement gives a meaningful throughput improvement for I/O-heavy workloads with near-zero code change — worth enabling by default in any production asyncio service.

Scaling

Horizontally scale stateless workers first — the gateway pattern in this module is designed to be replica-safe except for anything kept in-process (the in-process rate limiter from the Trade-offs section is exactly the kind of state that breaks this assumption at scale).
Autoscale on saturation signals, not CPU. Queue depth, semaphore wait time, and p99 latency are leading indicators of overload; CPU usage on an I/O-bound async service is often flat right up until it falls over.
Multi-process, shared-nothing for CPU-bound components (embedding generation, batch inference) — scale these as a separate worker fleet from the async request-handling gateway, not inside the same event loop.
Watch for exhaustion at the edges as concurrency grows — thread pools, database connection pools, and file descriptor limits all have ceilings that a purely “add more semaphore capacity” mindset will eventually hit.

Interview Questions

Walk me through what happens when you call time.sleep() inside an async function.

It blocks the entire event loop — every other coroutine scheduled on that loop stalls for the full duration, because time.sleep() is synchronous and gives control back to nothing. await asyncio.sleep() is the async-native equivalent that yields control back to the loop.

How would you bound concurrency to protect a downstream dependency?

A bounded asyncio.Semaphore sized to the downstream’s real capacity, acquired before the call and released in a finally block — exactly the _admit()/generate() pattern from this module’s Implementation section. Bonus depth: distinguish this from rate limiting (bounds arrival rate) — production systems usually need both, enforced at different points.

Why both, concretely

A semaphore limits how much work is in flight right now; a token bucket limits how fast new work is admitted over time. A burst of 1000 requests in one second with a semaphore of 32 and no rate limiter still queues all 1000 — the semaphore doesn’t reject anything, it just serializes access. Add a token bucket to reject excess arrivals outright instead of queueing them indefinitely.

How do you handle partial failures in a fan-out of concurrent calls?

Prefer asyncio.TaskGroup (structured concurrency) over bare asyncio.gather() when a failure in one task should cancel its siblings; use gather(..., return_exceptions=True) deliberately when partial results are acceptable and each result needs individual inspection — never let a single failed task’s exception silently vanish because it wasn’t awaited or inspected.

How do you shut a Python async service down safely?

Stop accepting new traffic (flip readiness false so the load balancer notices), drain in-flight requests with a bounded timeout, cancel background tasks, flush telemetry buffers, close clients and connection pools, then exit — the exact sequence the DrainManager in this module’s Production Example implements, with an explicit “did draining actually finish” signal rather than a silent kill.

Hands-on Lab

Every pattern in this module — bounded concurrency, deadlines, retry with jitter, circuit breaking, distributed rate limiting, graceful draining — is implemented and tested here, not described in the abstract. Read the full lab documentation →

labs/async-ai-gatewayproduction-ready

Extend the gateway's failure handling

Clone labs/async-ai-gateway and add a metric for queue-wait time, separate from total request latency (the exact distinction called out in this module’s Failure Modes section as “hidden queueing”). Write a test that asserts queue-wait time is reported even when the eventual provider call succeeds quickly — proving the metric captures admission delay, not just execution time.

Before calling an async Python service production-ready

Every remote call has an explicit timeout — connect, read, and total, not just “the default” - Concurrency is bounded with a semaphore sized to a measured downstream capacity, not a guess - Retries use capped exponential backoff with jitter, and respect a request-scoped deadline - Every finally block releases what its matching try acquired — semaphores, locks, connections - Cancellation (asyncio.CancelledError) is re-raised, never swallowed - p95/p99 latency, queue-wait time, and cancellation/timeout counts are tracked as metrics - Shutdown drains in-flight requests within a bounded timeout before the process exits

References

asyncio documentation, especially the Tasks and Streams sections.
PEP 3156 — the original asyncio proposal, still the clearest explanation of why the event loop is designed the way it is.
Nathaniel J. Smith, “Notes on structured concurrency, or: Go statement considered harmful”
Caleb Hattingh, Using Asyncio in Python (O’Reilly) — a concise, practical treatment of the concurrency models compared in this module’s Mental Model section.
uvloop documentation and its benchmarks against the standard library event loop.
labs/async-ai-gateway — every code example in this module, in its full, tested context.

Revision History

Version	Date	Change
1.0.0	2026-08-05	Initial publication, expanded from the retired static-prototype draft and grounded in the real async-ai-gateway lab code.