Shipping to conda-forge

Slides

What is a conda package (short version)?¶

A conda package is fundamentally a zip compressed archive (.conda) that contains two .tar.zst archives with one containing the package files and the other containing all the metadata. The JSON metadata describes the package structure, provenance, requirements, and dependencies. The package files consist of relocatable, platform-specific binaries and, optionally, symlinks (on supported platforms). The package format is simple yet general, powerful, and language agnostic — effectively any software can be packaged as a conda package.

Building a conda package¶

Conda packages are defined through a package recipe YAML file (recipe.yaml) with Jinja2 templating support and a robust schema. Recipes are built with a recipe build tool (rattler-build) into packages (the archives that are distributed).

The recipe file is separated into multiple sections that can conditionally be excluded or included

rattler-build can then be used to build the recipe at the command line with rattler-build’s CLI API:

# pixi global install rattler-build
rattler-build build --recipe <path to recipe directory>

Example conda recipes¶

Pure-Python (noarch: python)¶

(From the conda-forge maintainer documentation)

schema_version: 1

context:
  version: 1.2.3
  python_min: "3.10"

package:
  name: example-package
  version: ${{ version }}

source:
  url: https://pypi.org/packages/source/e/example-package/example_package-${{ version }}.tar.gz
  sha256: 12ff4785d337a1bb490bb7e9c2b1ee5da3112e94a8622f26a6c77f5d2fc6842a

build:
  noarch: python
  number: 0
  script: ${{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation

requirements:
  host:
    - python ${{ python_min }}.*
    - hatchling  # your build-system build tool goes here
    - pip
  run:
    - python >=${{ python_min }}

tests:
  - python:
      imports:
        - example_package
      python_version:
        - ${{ python_min }}.*
        - "*"
      pip_check: true

about:
  homepage: https://example.com/example-package
  license: BSD-3-Clause
  license_file: LICENSE
  summary: Single-line summary of the package.
  description: |
    One or two paragraphs with more information about the package.
  repository: https://github.com/example/example-package/
  documentation: https://example.com/example-package-docs/

extra:
  recipe-maintainers:
    # GitHub username
    - henryiii
    - matthewfeickert

Compiled extensions¶

If the package recipe has compiled extensions the package will be built against every version of Python that is currently supported by conda-forge.

schema_version: 1

context:
  version: 1.2.3

package:
  name: example-package
  version: ${{ version }}

source:
  url: https://pypi.org/packages/source/e/example-package/example_package-${{ version }}.tar.gz
  sha256: 12ff4785d337a1bb490bb7e9c2b1ee5da3112e94a8622f26a6c77f5d2fc6842a

build:
  number: 0
  script:
    env:
      CMAKE_GENERATOR: Ninja
    content:
      - ${{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation

requirements:
  build:
    - ${{ stdlib("c") }}
    - ${{ compiler("cxx") }}
    - cmake
    - ninja
    - if: build_platform != host_platform
      then:
        - cross-python_${{ host_platform }}
        - python
        - numpy
  host:
    - python
    - pip
    - scikit-build-core
    - numpy
  run:
    - python
    - numpy

tests:
  - python:
      imports:
        - example_package
      pip_check: true
  - script:
      - example-package-cli --help

about:
  homepage: https://example.com/example-package
  license: BSD-3-Clause
  license_file: LICENSE
  summary: Single-line summary of the package.
  description: |
    One or two paragraphs with more information about the package.
  repository: https://github.com/example/example-package/
  documentation: https://example.com/example-package-docs/

extra:
  recipe-maintainers:
    # GitHub username
    - henryiii
    - matthewfeickert

Submitting a package to conda-forge¶

The building of packages for the conda-forge conda channel is centralized through a community maintained build farm which builds all of the conda-forge feedstocks — GitHub repository with the conda package recipe, supporting scripts, and CI configuration — on https://github.com/conda-forge/. To get a conda recipe on conda-forge you need to submit it for build checks and review to https://github.com/conda-forge/staged-recipes.

Submitting recipes to conda-forge/staged-recipes workflow¶

1. Fork https://github.com/conda-forge/staged-recipes to your personal GitHub.

   # pixi global install gh git
   gh repo fork conda-forge/staged-recipes --clone=false
   git clone --depth 10 git@github.com:<your GitHub account username>/staged-recipes.git
   cd staged-recipes
   git remote add upstream git@github.com:conda-forge/staged-recipes.git
   git fetch --all --depth 10 --prune
   # You can optionally unshallow your clone if you want

2. Make a new branch from main for your package’s recipe.

   git checkout upstream/main -b feat/add-my-package

3. Create the package recipe

   • If the package exists on PyPI already use Pixi and grayskull to generate the recipe

     pixi run pypi <PyPI package name>

     Example:

     pixi run pypi boost-histogram

     This will produce the recipe at ./recipes/<PyPI package name>/.

   • If the package is not already on PyPI, copy the example rattler-build (“v1”) recipe into a directory named for your package (replacing example_package with your unique package name)

     cp -R ./recipes/example-v1 ./recipes/example_package

     and then edit the package’s recipe.yaml to meet your package needs following the hints in the comments, in the README.md, and the rattler-build docs/tutorials.

4. Once you have your recipe ready, stage and commit it to Git, and then lint the recipe

   pixi run lint

   If the linter fails, address the issues before continuing.

5. Run a test build of the package locally.

   pixi run <build-linux|build-osx|build-win>

6. If the test build passes, make sure all your changes are committed, and push your branch to your fork

   git push -u origin HEAD

   and then open a draft pull request against the upstream’s main branch.

   Ensure that you read, follow, and complete the checklist. Once your builds are passing, mark your PR as ready for review and tag the relevant conda-forge/staged-recipes review team. For pure-Python packages this would be

   @conda-forge/help-python, ready for review!

   and for packages with compiled extensions this would be

   @conda-forge/help-python-c, ready for review!

   Once you do so, the conda-forge-webservices will apply the relevant tags for the review team to find the recipe.

7. Once your recipe is reviewed and approved, it will be merged, which will then kick off automation to generate a conda-forge feedstock at https://github.com/conda-forge/<package name>-feedstock and to generate a maintainer team on GitHub populated by the extra.recipe-maintainers list in the package recipe. Your package feedstock will be automatically regenerated by the conda-forge-admin bot account and then the built packages will be automatically uploaded to https://anaconda.org/channels/conda-forge.

Getting help¶

When in doubt, just ask! The conda-forge Zulip server is the best way to get responses from the community and the conda-forge/core team, but also make sure to check out the maintainer knowledge base in the documentation first.

rattler-build recipe build example¶

Here is an valid rattler-build recipe for the collatz example from the A minimal compiled package with scikit-build section

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
schema_version: 1

context:
  version: 0.1.0

package:
  name: rattler-collatz
  version: ${{ version }}

source:
  url: https://github.com/scikit-build/SIMPLE-Py/archive/8c60b848fe7f8d72391d49c2130b27c88aad8dfb.tar.gz
  # pixi global install curl openssl
  # curl -sL <url> | openssl sha256
  sha256: fbd494e7216db678ec5b5f9f7bf7713faaf142690fd46298dcdd2e4d202dcc23

build:
  number: 0
  script:
    env:
      CMAKE_GENERATOR: Ninja
    content:
      - cd examples/5_02_pixi_build/compiled
      - ${{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation

requirements:
  build:
    # This is provided by conda-forge and so needs to be commented out for now.
    # - ${{ stdlib("c") }}
    - ${{ compiler("cxx") }}
    - cmake
    - ninja
    - if: build_platform != host_platform
      then:
        - cross-python_${{ host_platform }}
        - python
  host:
    - python
    - pip
    - scikit-build-core
    - pybind11
  run:
    - python

tests:
  - python:
      imports:
        - collatz
      pip_check: true

about:
  homepage: https://github.com/scikit-build/SIMPLE-Py
  license: BSD-3-Clause
  license_file: examples/5_02_pixi_build/compiled/LICENSE
  summary: Single-line summary of the package.
  description: |
    One or two paragraphs with more information about the package.
  repository: https://github.com/scikit-build/SIMPLE-Py
  documentation: https://github.com/scikit-build/SIMPLE-Py

extra:
  recipe-maintainers:
    # GitHub username
    - henryiii
    - matthewfeickert