25 — Packaging & Virtual Environments

Why Isolation Matters

pip install django==4.2
# later, a different project on the same machine:
pip install django==5.0
# without isolation, the SECOND install silently overwrites the first —
# both projects now share one global django version, and one of them breaks

Every Python installation has exactly one site-packages directory per interpreter — installing a package with pip (with no virtual environment active) writes into that shared, global location. Two projects on the same machine requiring different versions of the same dependency cannot coexist under a single global install; the second pip install simply replaces the first. Best practice: never run pip install against the system or global Python interpreter for project work — always activate a virtual environment first, so each project gets its own isolated site-packages.

venv: The Standard Library's Built-In Tool

python3 -m venv .venv          # creates a .venv/ directory with its own interpreter and site-packages
source .venv/bin/activate      # macOS/Linux
# .venv\Scripts\activate       # Windows (cmd.exe)
# .venv\Scripts\Activate.ps1   # Windows (PowerShell)

which python                    # now points INSIDE .venv/, not the system interpreter
python -m pip install requests  # installs into .venv/lib/.../site-packages, not globally

deactivate                       # restores the shell's original PATH

venv works by prepending the environment's bin/ (or Scripts/ on Windows) directory to PATH and pointing python/pip at copies (or symlinks) of the interpreter scoped to that directory — every pip install afterward lands in .venv/lib/pythonX.Y/site-packages instead of the system location. Best practice: name the directory .venv (the convention most tools, editors, and .gitignore templates already expect) and never commit it to version control — it's large, platform-specific, and trivially reproducible from a lockfile or requirements file.

# .gitignore
.venv/
__pycache__/
*.pyc
.pytest_cache/
.mypy_cache/

python -m pip vs bare pip

pip install requests
# WRONG in some setups — if a stray global `pip` executable is earlier on PATH
# than the venv's, this can install into the WRONG environment silently

python -m pip install requests
# CORRECT and unambiguous — always installs into whichever interpreter `python`
# currently resolves to, guaranteed to match the active venv

pip as a bare command is resolved via PATH lookup, which can point to an unexpected pip executable if a virtual environment wasn't activated cleanly (a common issue in CI scripts, subshells, or IDE-launched terminals). python -m pip instead runs pip as a module of the currently resolved python interpreter, removing that entire class of "installed into the wrong place" bugs. Best practice: use python -m pip install ... in scripts and CI, especially anywhere the active environment can't be visually confirmed.

requirements.txt: Simple, but Limited

pip freeze > requirements.txt

# requirements.txt — generated by pip freeze; PINS EVERY installed package, including transitive deps
certifi==2024.7.4
charset-normalizer==3.3.2
idna==3.7
requests==2.32.3
urllib3==2.2.2

python -m pip install -r requirements.txt

pip freeze dumps the entire environment — direct dependencies (requests) mixed indiscriminately with everything they pulled in transitively (urllib3, idna, certifi) — with no distinction between "I chose this" and "this got installed as a side effect." The core problem: requirements.txt has no concept of dependency groups (dev vs production vs test), no way to express version ranges vs exact pins in the same file cleanly, and no metadata about the project itself (name, version, entry points) — it's a flat list of pins, nothing more.

# requirements.txt — hand-written, direct dependencies only, loose version ranges
requests>=2.31,<3.0
click>=8.1

# requirements-dev.txt — a common convention to separate concerns
-r requirements.txt
pytest>=8.0
ruff>=0.5
mypy>=1.10

A common pre-pyproject.toml convention splits requirements.txt (production dependencies, loose ranges) from requirements-dev.txt (adds testing/linting tools, -r requirements.txt pulls in the base file) — workable, but entirely a hand-maintained convention with no tooling enforcing it, unlike the standardized structure pyproject.toml provides.

pyproject.toml: The Modern Standard

[project]
name = "orders-service"
version = "1.4.0"
description = "Order processing microservice"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "MIT" }
authors = [{ name = "Ada Lovelace", email = "ada@example.com" }]
dependencies = [
    "requests>=2.31,<3.0",
    "pydantic>=2.7",
    "click>=8.1",
]

[project.optional-dependencies]
dev = ["pytest>=8.0", "ruff>=0.5", "mypy>=1.10"]
docs = ["mkdocs>=1.6"]

[project.scripts]
orders-cli = "orders_service.cli:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

pyproject.toml (standardized by PEP 517/PEP 621) unifies what used to be scattered across setup.py, setup.cfg, and requirements.txt into one declarative file: project metadata, dependencies (with named optional groups like dev/docs, installed via pip install ".[dev]"), console-script entry points, and the build backend, all in a single tool-agnostic format that pip, build, poetry, uv, and every modern packaging tool understand.

pip install -e ".[dev]"
# -e : "editable install" — changes to source files take effect immediately,
#      no reinstall needed; standard for local development
# ".[dev]" : install the current directory's project, plus its "dev" optional group

Best practice: for any new project, start with pyproject.toml, not requirements.txt — it's the direction the entire ecosystem has moved (setup.py-based packaging is legacy as of recent setuptools/PyPA guidance), and tools like uv and poetry are built around it natively.

uv: The Fast Modern Toolchain

curl -LsSf https://astral.sh/uv/install.sh | sh

uv init orders-service               # scaffolds pyproject.toml + a starter layout
cd orders-service

uv add requests pydantic             # adds to pyproject.toml AND installs, resolving the whole tree
uv add --dev pytest ruff mypy        # adds to the "dev" dependency group specifically

uv run pytest                        # runs INSIDE the project's venv automatically — no activate needed
uv sync                               # installs exactly what uv.lock pins, reproducibly

uv python install 3.12                # uv can even manage Python interpreter versions itself
uv python pin 3.12

uv (from Astral, the makers of ruff) reimplements the entire pip/venv/pip-tools/pyenv toolchain in Rust, and is dramatically faster (often 10-100x) at dependency resolution and installation than pip, largely due to a global cache and a from-scratch resolver rather than pip's historically backtracking one. uv.lock is a fully resolved, hash-pinned lockfile (every transitive dependency, exact version, exact hash) — uv sync reproduces the exact same environment on any machine, unlike a loose requirements.txt range. Best practice: uv has rapidly become the recommended default for new Python projects as of 2025 — reach for it before pip+venv manually, or poetry, unless a project or team already has a specific reason to use something else.

poetry: The Established Alternative

curl -sSL https://install.python-poetry.org | python3 -

poetry new orders-service
cd orders-service

poetry add requests pydantic
poetry add --group dev pytest ruff mypy

poetry install                 # creates/updates the venv AND installs from poetry.lock
poetry run pytest              # runs a command inside poetry's managed venv
poetry shell                   # drops into an activated shell inside the venv

[tool.poetry]
name = "orders-service"
version = "1.4.0"
description = "Order processing microservice"

[tool.poetry.dependencies]
python = "^3.11"
requests = "^2.31"
pydantic = "^2.7"

[tool.poetry.group.dev.dependencies]
pytest = "^8.0"
ruff = "^0.5"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry predates uv by several years and popularized the "lockfile + dependency groups + one CLI for everything" workflow that uv later reimplemented for speed — poetry.lock serves the same reproducibility role as uv.lock. The trade-off today: poetry has a larger, more mature plugin ecosystem and longer production track record; uv is substantially faster and increasingly the default recommendation for new projects, but is newer and its ecosystem conventions are still stabilizing. Either is a legitimate choice; picking neither (bare pip + hand-written requirements.txt) is the option to avoid for anything beyond a throwaway script.

Publishing a Package to PyPI

pip install build twine

python -m build
# creates dist/orders_service-1.4.0-py3-none-any.whl (wheel — prebuilt, fast to install)
# and dist/orders_service-1.4.0.tar.gz (sdist — source distribution, buildable from scratch)

twine upload --repository testpypi dist/*     # ALWAYS rehearse on TestPyPI first
twine upload dist/*                            # the real, irreversible upload to PyPI

A wheel (.whl) is a prebuilt, ready-to-install artifact (no compilation step needed at install time, even for packages with C extensions, since the wheel is built per-platform); an sdist (.tar.gz) is the raw source, which pip falls back to building from scratch if no matching wheel exists for the installer's platform. Critical, unforgiving gotcha: PyPI does not allow re-uploading a file under a version number that was ever published before, even if it was deleted — a botched 1.4.0 upload cannot be fixed by re-uploading 1.4.0 again; the only path forward is bumping to 1.4.1. Best practice: always upload to TestPyPI first and verify pip install --index-url https://test.pypi.org/simple/ orders-service works end-to-end before touching the real index.

pip install twine
twine upload --repository testpypi dist/*
# Enter your API token (starts with pypi-) when prompted — NEVER your account password;
# PyPI deprecated password-based uploads in favor of scoped API tokens

Security best practice: authenticate with a scoped API token (generated per-project on PyPI's account settings page), not a username/password — a leaked project-scoped token can only publish that one package, while a leaked password compromises the entire account.

Semantic Versioning

# MAJOR.MINOR.PATCH
1.4.0 -> 1.4.1   # PATCH: backward-compatible bug fix
1.4.0 -> 1.5.0   # MINOR: backward-compatible new feature
1.4.0 -> 2.0.0   # MAJOR: breaking change — existing callers may need to update their code

[project]
dependencies = [
    "requests>=2.31,<3.0",   # accepts any 2.x >= 2.31, blocks the (potentially breaking) 3.0
    "pydantic~=2.7.0",       # ~= is "compatible release": accepts 2.7.x, blocks 2.8+
    "click==8.1.7",          # exact pin — no automatic updates at all, most reproducible, least flexible
]

Version specifiers in pyproject.toml are a policy decision, not a formality: >=2.31,<3.0 trusts the maintainer's semver promise that no 2.x release breaks the API; ==8.1.7 trusts nothing and pins exactly, trading flexibility for maximum reproducibility. Best practice: application pyproject.toml files (things you deploy, not libraries others depend on) should generally combine loose ranges in pyproject.toml with an exact-pinned lockfile (uv.lock/poetry.lock) for reproducible installs — libraries published to PyPI should keep ranges as loose as genuinely compatible, since an overly strict pin in a library needlessly constrains every downstream project that depends on it.

💡 Tips & Tricks

• Idiom: run python -m pip install --upgrade pip right after creating a fresh venv — the bundled pip version can lag behind the latest release for months, and newer pip versions resolve dependency conflicts more reliably.
• Debug: pip show <package> prints exactly where a package is installed from (Location:) — the fastest way to confirm whether an import is resolving to the virtual environment or an unexpected global install.
• Idiom: pip list --outdated (or uv pip list --outdated) shows every installed package with a newer version available — run it periodically rather than discovering a security fix was available three versions ago.
• Performance: uv's global package cache means installing the same package version across different projects' virtual environments is nearly instant after the first download — a meaningful speedup for anyone juggling many small projects or CI jobs.
• Safety: commit the lockfile (uv.lock, poetry.lock) to version control, but never the .venv/ directory itself — the lockfile is what makes an install reproducible across machines; the venv is a disposable, regeneratable artifact.

⚠️ Edge Cases & Gotchas

• A bare pip install with no active virtual environment silently writes into the global/system site-packages — on some Linux distributions this can even affect OS-level tools written in Python, which is precisely why modern pip versions refuse this by default (error: externally-managed-environment) unless a venv is active or --break-system-packages is explicitly passed.
• pip freeze > requirements.txt captures the environment as it happens to be right now, including packages installed for unrelated experimentation — running it in a venv that ever had a stray pip install some-debug-tool bakes that tool into the committed requirements file for everyone else on the team.
• PyPI permanently reserves every version number ever uploaded, even if deleted — there is no way to "fix" a bad 1.0.0 upload by re-uploading; the only forward path is a new version number, making a pre-upload TestPyPI rehearsal the only real safety net.
• ~= (compatible release) and >=,< ranges parse differently than most developers expect at the boundary: ~=2.7.0 means >=2.7.0, ==2.7.* (locks the minor version, only patch updates allowed), while ~=2.7 (no patch component) means >=2.7, ==2.* (allows minor updates too) — the number of version segments specified changes which segment is allowed to float.
• Editable installs (pip install -e .) historically wrote a .egg-link file and could behave inconsistently with namespace packages or certain build backends — modern pyproject.toml-based editable installs (PEP 660) are far more reliable, but an old-style setup.py-only editable install occasionally leaves the package importable from a stale path even after the source directory has moved.

🧠 Spot the Bug

A team commits requirements.txt to fix "works on my machine" issues, but a teammate's fresh clone still installs a different, incompatible version of a transitive dependency than the one used in production. Find the bug.

# requirements.txt, committed to the repo:
requests

# setup steps a new teammate follows:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# requirements.txt regenerated from the REAL, working environment:
pip freeze > requirements.txt
# certifi==2024.7.4
# charset-normalizer==3.3.2
# idna==3.7
# requests==2.32.3
# urllib3==2.2.2

uv add requests   # writes an exact, hash-pinned resolution into uv.lock
uv sync           # every teammate and CI run gets the IDENTICAL resolved tree

Key Takeaways

• Always work inside a virtual environment (venv, or one managed by uv/poetry) — installing directly into the global interpreter causes version conflicts between unrelated projects and, on some systems, is blocked outright.
• requirements.txt is a flat, unstructured list of pins with no concept of dependency groups or project metadata; pyproject.toml (PEP 621) is the modern, standardized replacement that unifies metadata, dependencies, optional groups, and build configuration.
• uv reimplements the whole toolchain in Rust for dramatic speed gains and is the increasingly-recommended default for new projects in 2025; poetry is the mature, established alternative with a longer track record.
• A lockfile (uv.lock, poetry.lock, or a pip freeze snapshot) pins the entire resolved dependency tree for true reproducibility — loose ranges in pyproject.toml alone do not guarantee two installs resolve identically.
• PyPI never allows reusing a version number once published, even after deletion — always rehearse a release on TestPyPI first, and authenticate with a scoped API token, never a password.
• Version specifiers are a policy choice: loose ranges (>=2.31,<3.0) suit published libraries that shouldn't over-constrain downstream users; exact pins suit deployed applications that need maximum reproducibility.