Skip to content
Principal AI Engineer Handbook

Python

Listen to this page6:47
Read the transcript

1. The machine underneath the language

Host: Welcome to the show. Today we’re taking apart Python — not the syntax, not the libraries, but the actual machine running underneath your code, because almost every weird bug or performance wall you hit in production traces back to one execution model. So let’s start with the blunt version: what is CPython actually doing when it runs your script?

Guest: At its core, CPython is a stack machine executing bytecode one instruction at a time, and critically, one thread at a time per interpreter — that’s the GIL, a mutex around the interpreter loop. On top of that, everything you touch is a heap-allocated object, even something as small as the integer 5, and its lifetime is managed by reference counting, with a generational cycle collector sitting behind it to catch reference loops that counting alone can’t free. Those two facts, single-threaded bytecode execution and everything-is-a-heap-object, explain a surprising amount of what feels mysterious about Python in practice.

Host: So if I’m building AI infrastructure and Python is gluing together the actual heavy lifting happening in C extensions, a database, or on a GPU, the message is: Python isn’t where the work happens, it’s the coordinator sitting on top of it. Over this episode we’re going to trace how that one model — single-threaded, refcounted, everything heap-allocated — shows up as gotchas in concurrency, memory, and the abstractions built on top of it.

2. The numbers that explain the surprises

Host: Okay, let’s get concrete, because I think people hear ‘reference counted’ and ‘single threaded’ and nod along without feeling the actual pain points. Give me the numbers that make those abstractions bite in real code.

Guest: Sure, start with something everyone’s tripped over: integers from negative five to 256 are cached, so ‘is’ comparisons on small ints just happen to work, and then you compare 257 to itself and it fails, because now you’ve got two distinct heap objects. Strings are always immutable, so building one with repeated plus-equals in a loop is quadratic — every append copies the whole string again — while ‘’.join() is linear because it allocates once. And floats are IEEE 754 doubles, so 0.1 plus 0.2 isn’t 0.3, it’s off in the seventeenth decimal place, which is exactly why money should live in Decimal or integer cents, not float dollars. Then there’s the boring one nobody thinks about until it bites: every Python object carries tens of bytes of overhead just for the refcount, type pointer, and bookkeeping, so a list of a million small ints is nowhere near four megabytes — it’s multiples of that.

Host: So the friendly high-level type system is quietly taxing you in bytes and CPU cycles the whole time. What about the recursion limit and multiprocessing — those feel like they belong in this same bucket of ‘default numbers you don’t notice until you do’.

Guest: Recursion defaults to a thousand frames, which sounds arbitrary but it’s there to turn a stack overflow into a catchable exception instead of a hard segfault — raise it and you’re just trading a clean crash for an uglier one. And multiprocessing’s start method quietly differs by platform: Linux forks the process by default, macOS and Windows spawn a fresh interpreter, and fork plus existing threads is a documented, reproducible way to deadlock. Miss that and you’ll debug a hang for a day before realizing it’s platform-specific behavior, not your code.

3. The bugs everyone eventually writes

Host: So let’s get concrete. If I hand you a junior engineer’s pull request, what’s the first bug you’re scanning for that traces back to everything we just described?

Guest: Mutable default arguments, every time — when a function’s default value is an empty list, that list gets evaluated once at definition, so every call without an argument shares the same object, and suddenly your ‘empty’ list has last request’s data in it. Late-binding closures are the sibling bug: a list comprehension of lambdas capturing a loop variable all return the final value, because the variable is looked up at call time, not creation time, and using ‘is’ where you meant equality shows up as an intermittent bug that only breaks on longer strings or bigger integers because small ones happen to be interned. Then there’s the shutdown class of bugs — bare except swallowing SystemExit and, since Python 3.8, CancelledError no longer being an Exception subclass, so code that catches Exception to be ‘safe’ now lets task cancellation slip through unnoticed, and __del__ running in undefined order at interpreter exit with its own exceptions silently discarded.

Host: And that’s before you even get to concurrency, where I assume threads not speeding up CPU-bound work and fork’s copy-on-write not being as free as people expect both live?

Guest: Exactly — threads add scheduling overhead to work the GIL already serializes, so you spawn four threads for a CPU-bound loop and it’s slower, not faster, than one. Fork copies memory lazily but bumps every refcount eagerly at fork time, so the moment a child touches those objects it dirties the page and copy-on-write buys you far less than the mental model promises. And module-level state — a cache or counter set at import time — is per-process, so workers silently stop sharing it, the exact same failure shape as an in-process rate limiter that doesn’t work across replicas.

4. Where the model actually bites in production

Host: So this stops being trivia the moment you’re picking an architecture. If someone’s choosing between threads, processes, and async for a real service, the GIL isn’t a footnote — it’s the constraint the whole design has to route around.

Guest: Right, and that’s exactly the shift we walk through in Module 1 on Production Python — concurrency model, layered timeouts, backpressure, graceful shutdown, all downstream of the mechanics we just covered. Pair that with the asyncio reference to see the runtime built on top of these semantics, and if you want it under exam pressure, the AI Systems Coding track puts these details directly in front of you. And when you’re ready to see it all assembled into something typed and tested, the async-ai-gateway lab is that service — same model, no longer a gotcha, just the architecture.

Generated from this page by Claude Sonnet 5 on , spoken by Kokoro-82M running locally. Two synthetic voices, not a recorded conversation. Every claim is drawn from this page — where it differs from the text above, the text is correct.

At a Glance

Section titled “At a Glance”

A dynamically typed, garbage-collected language whose reference implementation (CPython) executes bytecode on a stack machine, one thread at a time per interpreter. Almost every surprising performance characteristic of a Python service traces back to that last clause or to the fact that everything — including integers and functions — is a heap-allocated object.

For AI infrastructure the practical reading is narrow: Python is the coordination layer. The work that matters for throughput happens in C, in a database, or on an accelerator, and Python’s job is to orchestrate it without becoming the bottleneck itself.

Key Concepts

Section titled “Key Concepts”
Concept What it actually is
GIL A mutex around the interpreter. One thread executes Python bytecode at a time; it is released around blocking I/O and inside many C extensions
Reference counting The primary memory strategy. An object dies the moment its count hits zero — deterministic, unlike a tracing collector
Cycle collector The generational backup for reference cycles, which refcounting alone cannot free
__slots__ Replaces an instance __dict__ with fixed storage — meaningful memory savings on many small objects
Descriptor protocol What makes property, classmethod, and ORM fields work. __get__/__set__ on a class attribute
MRO (C3 linearization) The deterministic order in which multiple inheritance resolves attributes
Iterator vs generator Generators are lazy and single-pass; consuming one twice yields nothing the second time
Context manager __enter__/__exit__, the only construct that guarantees cleanup on the exception path
Type hints Erased at runtime by default. mypy --strict checks them; the interpreter does not
Virtual environment Directory-level dependency isolation. Not a sandbox and not a container
Wheel vs sdist A wheel is a prebuilt binary; an sdist compiles on install and is where “works on my machine” starts
GC generations Three, tuned for the observation that most objects die young

Numbers That Matter

Section titled “Numbers That Matter”
Quantity Value Why it matters
Threads running Python bytecode 1 per interpreter Threads help blocking I/O, never pure-Python CPU work
Recursion limit 1000 by default Deep recursion raises rather than segfaults; raising it trades one crash mode for another
Small-int cache −5 to 256 Why is appears to work on small integers and then fails at 257
Dict insertion order Guaranteed since 3.7 An implementation detail in 3.6 that became a language guarantee
str immutability Always Repeated += in a loop is quadratic; "".join() is linear
Default float IEEE 754 double 0.1 + 0.2 != 0.3; money belongs in Decimal or integer minor units
Object overhead Tens of bytes each A million-element list of ints is far larger than a million machine words
multiprocessing start method spawn on macOS and Windows, fork on Linux fork plus threads is a documented deadlock source
Import side effects Run once per interpreter Module-level code executes on first import, not on call

Common Gotchas

Section titled “Common Gotchas”
  • Mutable default arguments are evaluated once at definition. def f(x=[]) shares one list across every call — use None and build inside.
  • Late-binding closures. [lambda: i for i in range(3)] gives three functions all returning 2. Bind with a default argument: lambda i=i: i.
  • is is not ==. Identity, not equality. It appears to work for small integers and short strings because of interning, which makes the bug intermittent.
  • Bare except: catches SystemExit and KeyboardInterrupt — and since 3.8, asyncio.CancelledError no longer inherits from Exception, so except Exception behaves differently than older code assumed.
  • __del__ is not a destructor you can rely on. Order at interpreter shutdown is undefined, and an exception inside it is swallowed. Use a context manager.
  • Threads do not speed up CPU-bound Python. They add context-switching overhead to work the GIL already serializes. Use processes.
  • fork copies memory lazily but copies refcounts eagerly — touching objects in a child dirties pages, so copy-on-write saves less than expected.
  • Module-level state is per-process. A cache or counter defined at import time is not shared across workers, which is the same class of bug as an in-process rate limiter across replicas.
  • sys.setrecursionlimit does not grow the C stack. Push it far enough and the process segfaults rather than raising.

Where to Go Deeper

Section titled “Where to Go Deeper”