Pixi-build - SIMPLE-Py

Slides

In this section we’ll see how we can use Pixi Build to extend all of the packaging concepts that we’ve learned about so far into conda packages with as little additional effort as possible. Pixi Build allows for defining metadata in the Pixi manifest that define package, build, build-dependencies, host-dependencies, and run-dependencies TOML tables. These tables can be used together with the information in the rest of the Pixi manifest to build and install conda packages without having to have the full formal specification of a rattler-build recipe, like those shown in the Shipping to conda-forge section. To learn the sections and functionalities by example rather than by inspection of the spec, we’ll take the examples from previous packaging sections and adapt them to Pixi Build.

Basic Packaging¶

Configuring the Pixi manifest¶

Take the example Python package directory tree from Making a basic package as a starting foundation.

cp -R ./examples/5_02_pixi_build/basic /tmp/
mv /tmp/basic/original_pyproject.toml /tmp/basic/pyproject.toml
cd /tmp/basic

Next, create a Pixi manifest as a pyproject.toml [tool.pixi] table.

pixi init --format pyproject

which automatically creates environments from the optional dependencies and dependency groups.

✔ Added package 'rescale' as an editable dependency.
✔ Added environment 'dev' from optional dependencies or dependency groups.

pyproject.toml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "rescale"
version = "0.1.0"
description = "Rescale NumPy arrays to span [0, 1]."
readme = "README.md"
authors = [{ name = "My Name", email = "me@email.com" }]
license = "BSD-3-Clause"
license-files = ["LICENSE"]
keywords = ["arrays", "normalization"]
classifiers = [
  "Development Status :: 3 - Alpha",
  "Intended Audience :: Science/Research",
  "Programming Language :: Python :: 3",
  "Topic :: Scientific/Engineering",
  "Private :: Do Not Upload",
]
requires-python = ">=3.10"
dependencies = ["numpy>=1.24"]

[project.urls]
Homepage = "https://github.com/me/rescale"
"Bug Tracker" = "https://github.com/me/rescale/issues"
Changelog = "https://github.com/me/rescale/releases"

[dependency-groups]
dev = ["pytest"]

[tool.pixi.workspace]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tool.pixi.pypi-dependencies]
rescale = { path = ".", editable = true }

[tool.pixi.environments]
default = { solve-group = "default" }
dev = { features = ["dev"], solve-group = "default" }

[tool.pixi.tasks]

Now, enable the Pixi Build preview feature

preview = ["pixi-build"]

and add the platforms you might want to support (like normal)

pixi workspace platform add linux-64 osx-64 win-64 osx-arm64 linux-aarch64

change rescale from a Python package dependency to a conda package dependency

-[tool.pixi.pypi-dependencies]
-rescale = { path = ".", editable = true }
+[tool.pixi.dependencies]
+rescale = { path = "." }

add a [package] TOML table

[tool.pixi.package]
name = "rescale"
version = "0.1.0"

add the Python build backend to the [package.build.backend] table

[tool.pixi.package.build.backend]
name = "pixi-build-python"
version = "0.*"

add the host and run dependencies package tables

[tool.pixi.package.host-dependencies]
hatchling = "*"

[tool.pixi.package.run-dependencies]
numpy = ">=1.24"

and then add a tests task to the dev feature

pixi task add --feature dev tests "pytest tests"

resulting in the final pyproject.toml containing all the information for both a Python and conda package of rescale.

pyproject.toml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "rescale"
version = "0.1.0"
description = "Rescale NumPy arrays to span [0, 1]."
readme = "README.md"
authors = [{ name = "My Name", email = "me@email.com" }]
license = "BSD-3-Clause"
license-files = ["LICENSE"]
keywords = ["arrays", "normalization"]
classifiers = [
  "Development Status :: 3 - Alpha",
  "Intended Audience :: Science/Research",
  "Programming Language :: Python :: 3",
  "Topic :: Scientific/Engineering",
  "Private :: Do Not Upload",
]
requires-python = ">=3.10"
dependencies = ["numpy>=1.24"]

[project.urls]
Homepage = "https://github.com/me/rescale"
"Bug Tracker" = "https://github.com/me/rescale/issues"
Changelog = "https://github.com/me/rescale/releases"

[dependency-groups]
dev = ["pytest"]

[tool.pixi.workspace]
channels = ["conda-forge"]
platforms = ["linux-64", "osx-64", "win-64", "osx-arm64", "linux-aarch64"]
preview = ["pixi-build"]

[tool.pixi.dependencies]
rescale = { path = "." }

[tool.pixi.package]
name = "rescale"
version = "0.1.0"

[tool.pixi.package.build.backend]
name = "pixi-build-python"
version = "0.*"

[tool.pixi.package.host-dependencies]
hatchling = "*"

[tool.pixi.package.run-dependencies]
numpy = ">=1.24"

[tool.pixi.environments]
default = { solve-group = "default" }
dev = { features = ["dev"], solve-group = "default" }

[tool.pixi.feature.dev.tasks.tests]
cmd = "pytest tests"

Building (and installing) the package¶

To build and install the conda package into your environment run any Pixi command that requires an environment to be installed (such as pixi install, pixi run, pixi shell).

pixi run tests

You can see the build logs from Pixi Build and see that rescale is installed as a conda package

$ pixi list -x
Installed for: linux-64
Name     Version  Build                    Size  Kind   Source
numpy    2.5.1    py314h2b28147_0      8.67 MiB  conda  https://conda.anaconda.org/conda-forge
python   3.14.6   habeac84_100_cp314  35.02 MiB  conda  https://conda.anaconda.org/conda-forge
rescale                                          conda  .

Building a conda package¶

We can use pixi publish to build the package into a conda package and either publish it to a (--target-channel) conda channel or copy the artifact into a (--target-dir) local directory.

pixi publish --clean --target-dir .

📦 Publishing 1 package(s) to directory </path/to/cwd>
✔ Successfully published 1 package(s) to directory </path/to/cwd>
  - rescale-0.1.0-pyh4616a5c_0.conda

We can use rattler-build to inspect the package and get more information about it

pixi global install rattler-build
rattler-build package inspect ./*.conda

Exercise 1 (Installing from local conda channels)

Above we mentioned that pixi publish can publish to a conda channel. Publish the package to a local conda channel on your filesystem and then install the rescale Pixi package in a new Pixi workspace.

Solution to Exercise 1

First publish the conda package to a local channel on the file system.

pixi publish --clean --target-channel /tmp/local-channel

Then create a new Pixi workspace and add the package

pixi init /tmp/local-channel-example && cd /tmp/local-channel-example
pixi workspace channel add --prepend "/tmp/local-channel"
pixi add rescale
pixi list rescale

Exercise 2 (Publish to a personal conda channel)

Publish the conda package to a personal prefix.dev conda channel.

Solution to Exercise 2

If you have a personal conda channel on prefix.dev you can publish with

pixi publish --clean --target-channel https://prefix.dev/<channel-name>

Compiled Packaging¶

Take the example Python package directory tree from the A minimal compiled package with scikit-build section as a starting foundation.

cp -R ./examples/2_01_package/collatz /tmp/compiled
cd /tmp/compiled

Next, create a Pixi manifest as a pyproject.toml [tool.pixi] table.

pixi init --format pyproject

pyproject.toml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build"

[project]
name = "collatz"
version = "0.1.0"

[tool.pixi.workspace]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tool.pixi.pypi-dependencies]
collatz = { path = ".", editable = true }

[tool.pixi.tasks]

Now, enable the Pixi Build preview feature

preview = ["pixi-build"]

and add the platforms you might want to support (like normal)

pixi workspace platform add linux-64 osx-64 win-64 osx-arm64 linux-aarch64

change collatz from a Python package dependency to a conda package dependency

-[tool.pixi.pypi-dependencies]
-collatz = { path = ".", editable = true }
+[tool.pixi.dependencies]
+collatz = { path = "." }

add a [package] TOML table

[tool.pixi.package]
name = "collatz"
version = "0.1.0"

add the Python build backend to the [package.build.backend] table.

[tool.pixi.package.build.backend]
name = "pixi-build-python"
version = "0.*"

As we are using compilers we need to tell Pixi Build the compiler types that we require, and it can figure out the rest. We can also set ignore-pypi-mapping to automatically enable PyPI-to-conda name mapping (which somewhat confusingly happens under false) to not have to redefine the host dependencies.

[tool.pixi.package.build.config]
# c.f. https://pixi.prefix.dev/latest/build/backends/pixi-build-python/#compilers
# c.f. https://pixi.prefix.dev/latest/build/backends/pixi-build-python/#ignore-pypi-mapping
compilers = ["cxx"]
ignore-pypi-mapping = false  # Enable automatic PyPI-to-conda mapping

...

# Unneeded given package.build.config.ignore-pypi-mapping = false
# [tool.pixi.package.host-dependencies]
# scikit-build-core = "*"
# pybind11 = "*"

Then add the the necessary build requirements.

[tool.pixi.package.build-dependencies]
cmake = "*"
ninja = "*"

Finally, we want to take advantage of scikit-build-core’s defaults of using Ninja. rattler-build (through pixi-build-python) exports CMAKE_GENERATOR='Unix Makefiles' as a default behavior, which overrides scikit-build-core’s default of Ninja.^[1] To restore the default behavior provide scikit-build-core explicit arguments.

[tool.scikit-build]
# c.f. https://github.com/prefix-dev/rattler-build/issues/2487
cmake.args = ["-GNinja"]

pyproject.toml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
[build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build"

[project]
name = "collatz"
version = "0.1.0"

[tool.pixi.workspace]
channels = ["conda-forge"]
platforms = ["linux-64", "win-64", "osx-arm64", "linux-aarch64"]
preview = ["pixi-build"]

[tool.scikit-build]
# c.f. https://github.com/prefix-dev/rattler-build/issues/2487
cmake.args = ["-GNinja"]

[tool.pixi.dependencies]
collatz = { path = "." }

[tool.pixi.package]
name = "collatz"
version = "0.1.0"

[tool.pixi.package.build.backend]
name = "pixi-build-python"
version = "0.*"

[tool.pixi.package.build.config]
# c.f. https://pixi.prefix.dev/latest/build/backends/pixi-build-python/#compilers
# c.f. https://pixi.prefix.dev/latest/build/backends/pixi-build-python/#ignore-pypi-mapping
compilers = ["cxx"]
ignore-pypi-mapping = false

[tool.pixi.package.build-dependencies]
cmake = "*"
ninja = "*"

# Unneeded given package.build.config.ignore-pypi-mapping = false
# [tool.pixi.package.host-dependencies]
# scikit-build-core = "*"
# pybind11 = "*"

[tool.pixi.tasks]

So in just over 20 lines of TOML, we can build a conda package for a Python package with compiled extensions!

Exercise 3 (Build and install the conda package)

Build collatz as a conda package and install it into your Pixi workspace environment.

Solution to Exercise 3

pixi install
pixi run python -c 'from collatz import collatz_steps; print(collatz_steps(27))'

You can verify with

pixi list -x

Exercise 4 (Build a conda package archive)

Build collatz as a local conda package archive and publish it to a local channel and inspect it.

Solution to Exercise 4

pixi publish --clean --target-channel /tmp/local-channel
rattler-build package inspect /tmp/local-channel/linux-64/collatz*.conda

Exercise 5 (Install collatz as a conda package from source)

A full example is available at https://github.com/scikit-build/SIMPLE-Py/tree/main/examples/5_02_pixi_build/compiled. Add collatz as a source dependency to a Pixi workspace and install it as a conda package.

Solution to Exercise 5

pixi init git-source-pixi-build && cd git-source-pixi-build

[workspace]
channels = ["conda-forge"]
name = "git-source-pixi-build"
platforms = ["linux-64"]
version = "0.1.0"
preview = ["pixi-build"]

[tasks]

[dependencies]

[dependencies.collatz]
git = "https://github.com/scikit-build/SIMPLE-Py"
branch = "main"
subdirectory = "examples/5_02_pixi_build/compiled"

Exercise 6 (Install a conda package from a new channel)

collatz was published on the conda channel https://prefix.dev/matthewfeickert.

pixi auth login --token <token> prefix.dev
pixi publish --clean --target-channel https://prefix.dev/matthewfeickert

Create a new workspace and install from that channel.

Solution to Exercise 6

pixi init prefix-channel-example && cd prefix-channel-example
pixi workspace channel add https://prefix.dev/matthewfeickert
pixi add collatz
pixi list -x

Footnotes¶

c.f. prefix-dev/rattler-build#2487
↩