A minimal compiled package with scikit-build

Slides

You’ve seen how to build a package already. Let’s try a compiled package now!

First simple package¶

First, you’ll need a file to compile. C++, C, Fortran, Rust, etc. You also need to pick a binding tool; if you are in C, you can use the C-API, but don’t -- it’s verbose, and you’ll have to manage reference counting; binding tools handle this for you.

Let’s use a trivial pybind11 extension. Name the source file collatz.cpp:

#include <pybind11/pybind11.h>
#include <stdexcept>

// Even -> n/2, odd -> 3n+1. Conjectured to always terminate.
int collatz_steps(long long n) {
  if (n < 1) throw std::domain_error("n must be >= 1");
  int steps = 0;
  while (n != 1) {
    n = (n % 2 == 0) ? n / 2 : 3 * n + 1;
    ++steps;
  }
  return steps;
}

PYBIND11_MODULE(collatz, m) {
  m.def("collatz_steps", &collatz_steps, "Steps to reach 1 in the Collatz sequence");
}

It’s a tight integer loop, something that’s fast in a compiled language. Hopefully you noticed that we set a module name, collatz, in PYBIND11_MODULE. Every CPython module needs to know the name it will be compiled to, since that’s also how you look up the main entry point into the code. The sequence is only defined for positive integers, so the guard rejects n < 1 — pybind11 maps std::domain_error to a Python ValueError.

pybind11 isn’t the only choice. nanobind is a lighter, faster tool from the same author, and it also builds with CMake, so everything below works with a one-line swap. The binding itself looks almost identical:

#include <pybind11/pybind11.h>

PYBIND11_MODULE(collatz, m) {
  m.def("collatz_steps", &collatz_steps);
}

#include <nanobind/nanobind.h>

NB_MODULE(collatz, m) {
  m.def("collatz_steps", &collatz_steps);
}

We’ll stick with pybind11 for this chapter. The next chapter compares the binding tools in depth, and the following chapter covers Rust via PyO3; here we care about the packaging around them, which is the same either way.

Now, you need a pyproject.toml. We use scikit-build-core as the build backend, and list pybind11 so CMake can find it while building:

[build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build"

[project]
name = "collatz"
version = "0.1.0"

And, a CMakeLists.txt:

cmake_minimum_required(VERSION 3.15...4.3)
project(collatz LANGUAGES CXX)

find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(collatz collatz.cpp)
install(TARGETS collatz DESTINATION .)

And that should be it! Before you run it, take a guess: starting from 27, how many steps do you think the sequence takes to reach 1? (The number climbs to 9232 on the way, so don’t feel bad if you’re off.) Now try it:

$ uv run python
>>> from collatz import collatz_steps
>>> collatz_steps(27)
111

$ uv run python -c "from collatz import collatz_steps; print(collatz_steps(27))"
111

Was it worth it?¶

We claimed a tight integer loop is “fast in a compiled language.” Let’s check. Here’s the same function in pure Python:

def collatz_steps(n):
    steps = 0
    while n != 1:
        n = n // 2 if n % 2 == 0 else 3 * n + 1
        steps += 1
    return steps

timeit makes the comparison easy — point it at each version and let it pick a sensible number of loops:

$ python -m timeit -s "from collatz import collatz_steps" "collatz_steps(97)"
1000000 loops, best of 5: 210 nsec per loop

$ python -m timeit -s "from mymodule import collatz_steps" "collatz_steps(97)"
100000 loops, best of 5: 5.8 usec per loop

Exact numbers depend on your machine, but the shape holds: the compiled version is roughly 20–30× faster here. That gap is the whole reason to reach for a compiled extension — and it’s why the loop lives in C++ while the packaging, which runs once, stays in Python.

Make the package better¶

You don’t have to add anything more, but there are config settings that really help, let’s look at a few.

Minimum version¶

How do you ensure new users get good defaults, but old users don’t break when defaults change? CMake handles this elegantly:

cmake_minimum_required(VERSION 3.15)

That 3.15 is special: it’s the minimum version of CMake supported. It doesn’t just add an error if CMake is too old, though; it actually changes the defaults (called Policies). It even handles removals; you can still use FindPythonInterp/FindPythonLibs if it’s set below 3.26, but if it’s 3.27 or higher, those modules are no longer available.

(There’s also an optional upper number: that lets the minimum version float up to the upper number, based on the CMake that’s running. Using it extends the lifespan of your code, but does mean you should test once on either end of the range. Update it as you test on newer versions.)

Scikit-build-core adopts a similar mechanism:

[tool.scikit-build]
minimum-version = "1.0"

Any changes we make are gated behind this version; if you set it, then we will continue to behave identically. But if you increase it (or leave it unset), the behavior will change to the latest recommendations. For example, we improved the SDist inclusion mode to make it more useful and faster, but if you set something older than "0.12" here, you’ll get the old mode.

If we have to change something due to PyPI or some other tool we don’t control (like non-normalized SDist names no longer being uploadable on PyPI), then that won’t be gated by this value.

Since you should also provide a minimum in your build system requirements, there’s a great trick you can use to avoid repeating yourself:

[build-system]
requires = ["scikit-build-core>=1.0"]
build-backend = "scikit_build_core.build"

[tool.scikit-build]
minimum-version = "build-system.requires"

Setting it to this special string will read the value from the requires list.

Structuring the files¶

Just like pure Python packages (though probably even more important), you can use src layout:

example
├── pyproject.toml
├── CMakeLists.txt
└── src
    └── example
        ├── __init__.py
        └── _core.cpp

This ensures that Python doesn’t pick up the local folder if you run import example inside the project directory. This is especially important for compiled code, since you (and your tests, etc) can’t run the uncompiled version!

Just like hatchling (which we took a lot of inspiration from!), the name matters! Your package name must match the project name, and be in /, /src, or /python. If not, you need to tell it where to discover it:

[tool.scikit-build]
wheel.packages = ["some/path/to/package"]

The final directory name is the package. If you need more complex structure, a table is supported here as well.

example
├── pyproject.toml
├── CMakeLists.txt
└── src
    └── collatz
        ├── __init__.py
        └── _core.cpp

PYBIND11_MODULE(_core, m) {
  m.def("collatz_steps", &collatz_steps, "Steps to reach 1 in the Collatz sequence");
}

from ._core import collatz_steps

__all__ = ["collatz_steps"]

find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(_core src/collatz/_core.cpp)
install(TARGETS _core DESTINATION collatz)

The two distributions¶

A package ships as two artifacts, and it’s worth seeing how they relate before we build each one. The SDist is the source you build from; the wheel is the pre-built result you install:

The two differ in what they contain: an SDist keeps build inputs (source, CMakeLists.txt, tests), while a wheel keeps only the runtime package plus the compiled extension. Deciding what lands in each is the tricky part, so let’s take them one at a time.

Building the SDist¶

An SDist is the source of your package, and it needs to contain everything required to build a wheel — but everything required is surprisingly hard to pin down automatically. This is one of the main differences between backends.

Scikit-build-core and hatchling start by including anything not .gitignored, and they include your .gitignore. That’s a good baseline, doesn’t require git, and enables sdists to recreate themselves if unpacked. Then there are other ways to add/remove files (and scikit-build-core includes other modes to pick from).

flit-core includes a few common files and your package, and otherwise requires explicit includes/excludes. Setuptools is similar, but uses MANIFEST.in.

To build an sdist:

uv build --sdist

To see what’s in your SDist, use tar -tf dist/*.tar.gz. To measure what’s in your SDist against git as a source of truth, you can use uvx check-sdist. It’s really useful for flit-core/setuptools, less so for scikit-build-core and hatchling, due to the fact they tend to map to git by default.

If you need files from somewhere else, you can use sdist.force-include (spelling slightly different in hatchling).

Building the wheel¶

Figuring out what files go in the wheel is also a hard problem, though at least it’s better defined; only your package goes in, tests, docs, etc. do not.

Scikit-build-core includes everything in the auto-discovered or explicitly named wheel.packages. It also contains anything CMake installs.

You can adjust quite a bit, though. For example, wheel.install-dir sets where CMake’s install tree is grafted into the wheel, so you can keep DESTINATION . in CMake and redirect everything from one place:

[tool.scikit-build]
wheel.install-dir = "collatz"

install(TARGETS _core DESTINATION .)

Now the target installed to . lands in collatz/ inside the wheel. Be careful: paths here are relative to the wheel root, and an absolute path is an error.

You can also use wheel.force-include to move things around, and wheel.exclude to strip out items you don’t want. There are controls over what “components” CMake installs, which can allow you to pick a subset from CMake.

To build a wheel:

uv build --wheel

If you leave off the flags, it builds both, by the way. Unlike python -m build — which builds the SDist and then builds the wheel from that SDist — uv build builds the two independently, both directly from the source directory. If you want to build a wheel from an SDist (a good check that your SDist is complete), pass the archive, such as: uv build dist/collatz-0.1.0.tar.gz.

To see what’s in your wheel, use unzip -l dist/*.whl.

Iterating: the full compiled edit loop¶

So far each change meant a fresh build. For real development you want an editable install so Python picks up your package in place — and for compiled code, scikit-build-core can even rebuild the extension automatically when you re-import it. That’s covered in its own chapter; the short version is:

uv pip install --no-build-isolation -e .

When a build goes wrong, two knobs help you see what happened:

[tool.scikit-build]
build.verbose = true      # show the full compiler command lines
cmake.build-type = "Debug"

You can also pass CMake defines through the build without editing files, which is handy for one-off experiments:

uv build --wheel -C cmake.define.CMAKE_CXX_STANDARD=20

long long collatz_max(long long n) {
  long long peak = n;
  while (n != 1) {
    n = (n % 2 == 0) ? n / 2 : 3 * n + 1;
    peak = n > peak ? n : peak;
  }
  return peak;
}

PYBIND11_MODULE(_core, m) {
  m.def("collatz_steps", &collatz_steps);
  m.def("collatz_max", &collatz_max);
}

Glossary¶