Setting up for development

Slides

Before you begin¶

You’ll want to have the following tools installed:

• uv: A fast package manager written in Rust that replaces pip, venv, pipx, and more.

• pixi: A fast package manager written in Rust for conda packages (high-level only replacement for conda/mamba)

• A system compiler for the later parts of the workshop. (xcode-clt for macOS, Visual Studio for Windows, GCC for Linux)

These tools will be handy too, but they can be all installed via uv tool install so you don’t need to pre-install them:

• prek: A linter/formatter runner, written in Rust (similar to pre-commit)

• nox: A task runner

Low level concepts¶

(Or how not to break your computer)

The topics in this session are there just to give you an understanding of what’s going on. We’ll introduce high level abstractions in the next section!

Virtual environments¶

The first thing you should know when packaging is how to install stuff. There are three options:

A system or user install sounds nice, but you can’t install incompatible packages, they can break your system, you can’t control what each project needs, you can end up not knowing what your requirements are, it’s hard to update, etc.

The standard solution is a virtual environment. It places files inside a folder with a name you choose (.venv in the project folder is the standard choice) that looks like this:

.venv
├── .gitignore      # These make sure tools know not to include/archive
├── CACHEDIR.TAG    # this folder
│
├── bin
│   ├── activate    # "activation" scripts for different shells (bash default)
│   ...
│   ├── python      # Symlinks to your local Python install
│   ├── python3
│   └── python3.14
│
├── lib
│   └── python3.14
│       └── site-packages  # This is where installed libraries go
│           ...
│
└── pyvenv.cfg      # Special file telling Python this is a virtual environment

When Python runs, it checks to see if there’s a pyvenv.cfg above it. If there is, it knows it is in a virtual environment (venv) and reads site-packages from there. There are two ways to use it:

.venv/bin/python ...

. .venv/bin/activate  # Shell specific
python ...
deactivate

The . at the start (most shells support source as well) allows the activation script to set environment variables, something a normal application could not do. It sets PATH (and VIRTUALENV, but that is informational) so things inside the virtual environment’s bin are at the beginning of the PATH.

To create one of these, you have several options:

python3 -m venv .venv

virtualenv .venv

uv venv

We will be using uv, which can do a lot of this for us.

Requirements¶

Now that you know how to make virtual environments, how should you install stuff into them? If the environment is active, then installers will target it. If an environment is not active, uv pip install will check for a .venv folder, and will target that by default (which is why it doesn’t need to install pip in the venv).

But a virtual environment is meant to be expendable. You should be able to delete it and recreate it any time. So instead of manually installing, you want to list packages in some format:

Locked dependencies means that every dependency is fully specified, ideally with a SHA to make sure users get exactly the versions you used. Using a lockfile is a great way to recover a virtual environment exactly on another machine. However, dependencies cannot be locked for libraries; it would be a problem if your two favorite libraries could not be installed together in one environment because they conflicted on pins!

Summary¶

So far, we have discussed:

This might seem simple, but creating, activating, installing, and locking are all separate steps with different incantations and different tools.

uv venv                 # defaults to .venv
uv pip install cowsay   # defaults to .venv
.venv/bin/cowsay        # run without activation is fine
rm -r .venv             # venvs can be removed

High level packaging¶

Apps¶

Persistent apps¶

Let’s break up applications and libraries. An application is something you install and run, but (assuming you use virtual environments) never needs to be installed with other unrelated things (apps that support plugins are okay). Generally, you won’t import an app inside Python, you’ll run it from the command line (or a graphical interface, etc).

This special property allows us to do something interesting. Imagine we:

1. Made a venv somewhere

2. Installed our application in it

3. Put just that application somwehere on our PATH

Since we never need to import it, we can get away without activation. This is exactly what pipx (pip for executables) and uv tool do!

Use uv tool install to install apps. Use uv tool list to see what you’ve installed. uv tool upgrade to upgrade, and uv tool uninstall to remove them. (See uv tool --help).

uv tool install cowsay
cowsay
uv tool uninstall cowsay

Single use apps¶

We can do one better if we have something we don’t want to run all the time. The install and run steps can be combined! This is such a common need that uv comes with a separate CLI uvx that does this. Running this:

uvx cowsay

Will make a venv, download the app, then run a command with the same name in the venv. If you run it again, it will recreate the venv if it’s over a week old.

With this, you basically have all of PyPI at your fingertips, and you don’t have to remember to update things too!

Single file scripts¶

Another thing we can do is single-file scripts. They look like this:

# /// script
# dependencies = ["numpy"]
# ///

import numpy as np

if __name__ == "__main__":
    print(np.array([1, 2, 3]))

When you run it:

uv run single.py

pipx run single.py

The dependencies will be downloaded into a temporary venv.

Projects (websites, etc)¶

These are very similar to libraries (below); the key difference is you should always commit your lockfile (vs. being optional for libraries).

How you specify your dependencies depends a bit on the tool you are using, since it is not standardized like libraries are. But you can always use [project] and make it an unpublishable library by adding this:

[project]
classifiers = [
  "Private :: Do Not Upload",
]

Libraries¶

A library is something that can be shared with others. The key feature is that it must be able to share an environment with whatever else the user of the library needs. If a user is expected to import your code, it’s a library.

Later sections will explain a lot more about packages, so let’s just present a basic pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "example"
version = "0.1.0"

Notice the [build-system] section; this tells tools how to build the package into something you can distribute and install. There are quite a few build backends; the one above uses hatchling.

There are a lot of places to put dependencies; here’s an expanded version with annotations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[build-system]
requires = ["hatchling"]  # 1
build-backend = "hatchling.build"

[project]
name = "example"
version = "0.1.0"
dependencies = [] # 2

[project.optional-dependencies]
extra = []  # 3

[dependency-groups]
dev = [] # 4

A quick comparison:

High level project management with uv¶

When dependency-groups were introduced, uv made a brilliant decision: if there’s a group named dev, it is installed by default when using uv run. This enables the following file:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "example"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["numpy"]

[dependency-groups]
dev = ["pytest"]

to be all you need to use uv run. For example,

uv run pytest

will:

• Download Python if you don’t have a valid copy

• Make a virtual environment at .venv if it doesn’t exist

• Install dependencies from a uv.lock lockfile if it exists

• Create a uv.lock file from project.dependencies and dependency-groups.dev if it doesn’t exist and install

• Make an editable install of your project

• Run whatever command you give it from that virtual environment

If you edit the dependencies, then the lockfile and .venv will be updated when you uv run again.

Any command works, so uv run python starts up python, if you have command line apps you can use uv run ..., etc.

git clone https://github.com/pypa/packaging
cd packaging
uv run pytest