Building extension modules

Building Extension Modules #

Poetry allows a project developer to introduce support for, build and distribute native extensions within their project. In order to achieve this, at the highest level, the following steps are required.

1. Add Build Dependencies

   The build dependencies, in this context, refer to those Python packages that are required in order to successfully execute your build script. Common examples include cython, meson, maturin, setuptools etc., depending on how your extension is built.

   Note

   You must assume that only Python built-ins are available by default in a build environment. This means, if you need even packages like setuptools, it must be explicitly declared.

   The necessary build dependencies must be added to the build-system.requires section of your pyproject.toml file.

   [build-system]
   requires = ["poetry-core", "setuptools", "cython"]
   build-backend = "poetry.core.masonry.api"
   

   Note

   It is recommended that you consider specifying version constraints to all entries in build-system.requires in order to avoid surprises if one of the packages introduce a breaking change. For example, you can set cython to cython>=3.0.11,<4.0.0 to ensure no major version upgrades are used.
   Note

   If you wish to develop the build script within your project’s virtual environment, then you must also add the dependencies to your project explicitly to a dependency group - the name of which is not important.

   poetry add --group=build setuptools cython
   

2. Add Build Script

   The build script can be a free-form Python script that uses any dependency specified in the previous step. This can be named as needed, but must be located within the project root directory (or a subdirectory) and also must be included in your source distribution. You can see the example snippets section for inspiration.

   Note

   The build script is always executed from the project root. And it is expected to move files around to their destinations as expected by Poetry as per your pyproject.toml file.

   [tool.poetry.build]
   script = "relative/path/to/build-extension.py"
   

   Note

   The name of the build script is arbitrary. Common practice has been to name it build.py, however this is not mandatory. You should consider using a subdirectory if feasible.

3. Specify Distribution Files

   Warning

   The following is an example, and should not be considered as complete.

   [tool.poetry]
   ...
   include = [
       { path = "package/**/*.so", format = "wheel" },
       # sources must be present in sdist, can be ignored if you only have *.pyx sources
       { path = "package/**/*.c", format = "sdist" },
   ]
   

   The key takeaway here should be the following. You can refer to the pyproject.toml documentation for information on each of the relevant sections.

   1. Include your build outputs in your wheel.
   2. Exclude your build inputs from your wheel.
   3. Include your build inputs to your source distribution.

Example Snippets #

Cython #

[build-system]
requires = ["poetry-core", "cython", "setuptools"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
...
packages = [
    { include = "package", from = "src"},
]
include = [
    { path = "src/package/**/*.so", format = "wheel" },
]
# if not already excluded via .gitignore
exclude = [
    "**/*.c"
]

[tool.poetry.build]
script = "scripts/build-extension.py"

from __future__ import annotations

import os
import shutil

from pathlib import Path

from Cython.Build import cythonize
from setuptools import Distribution
from setuptools import Extension
from setuptools.command.build_ext import build_ext


COMPILE_ARGS = ["-march=native", "-O3", "-msse", "-msse2", "-mfma", "-mfpmath=sse"]
LINK_ARGS = []
INCLUDE_DIRS = []
LIBRARIES = ["m"]


def build() -> None:
    extensions = [
        Extension(
            "*",
            ["src/package/*.pyx"],
            extra_compile_args=COMPILE_ARGS,
            extra_link_args=LINK_ARGS,
            include_dirs=INCLUDE_DIRS,
            libraries=LIBRARIES,
        )
    ]
    ext_modules = cythonize(
        extensions,
        include_path=INCLUDE_DIRS,
        compiler_directives={"binding": True, "language_level": 3},
    )

    distribution = Distribution({
        "name": "package",
        "ext_modules": ext_modules
    })

    cmd = build_ext(distribution)
    cmd.ensure_finalized()
    cmd.run()

    # Copy built extensions back to the project
    for output in cmd.get_outputs():
        output = Path(output)
        relative_extension = Path("src") / output.relative_to(cmd.build_lib)

        shutil.copyfile(output, relative_extension)
        mode = os.stat(relative_extension).st_mode
        mode |= (mode & 0o444) >> 2
        os.chmod(relative_extension, mode)


if __name__ == "__main__":
    build()

scripts/
└── build-extension.py
src/
└── package
    ├── example.pyx
    └── __init__.py

Meson #

[tool.poetry.build]
script = "build-extension.py"

[build-system]
requires = ["poetry-core", "meson"]
build-backend = "poetry.core.masonry.api"

from __future__ import annotations

import subprocess

from pathlib import Path


def meson(*args):
    subprocess.call(["meson", *args])


def build():
    build_dir = Path(__file__).parent.joinpath("build")
    build_dir.mkdir(parents=True, exist_ok=True)

    meson("setup", build_dir.as_posix())
    meson("compile", "-C", build_dir.as_posix())
    meson("install", "-C", build_dir.as_posix())


if __name__ == "__main__":
    build()

Maturin #

[tool.poetry.build]
script = "build-extension.py"

[build-system]
requires = ["poetry-core", "maturin"]
build-backend = "poetry.core.masonry.api"

import os
import shlex
import shutil
import subprocess
import zipfile

from pathlib import Path


def maturin(*args):
    subprocess.call(["maturin", *list(args)])


def build():
    build_dir = Path(__file__).parent.joinpath("build")
    build_dir.mkdir(parents=True, exist_ok=True)

    wheels_dir = Path(__file__).parent.joinpath("target/wheels")
    if wheels_dir.exists():
        shutil.rmtree(wheels_dir)

    cargo_args = []
    if os.getenv("MATURIN_BUILD_ARGS"):
        cargo_args = shlex.split(os.getenv("MATURIN_BUILD_ARGS", ""))

    maturin("build", "-r", *cargo_args)

    # We won't use the wheel built by maturin directly since
    # we want Poetry to build it but, we need to retrieve the
    # compiled extensions from the maturin wheel.
    wheel = next(iter(wheels_dir.glob("*.whl")))
    with zipfile.ZipFile(wheel.as_posix()) as whl:
        whl.extractall(wheels_dir.as_posix())

        for extension in wheels_dir.rglob("**/*.so"):
            shutil.copyfile(extension, Path(__file__).parent.joinpath(extension.name))

    shutil.rmtree(wheels_dir)


if __name__ == "__main__":
    build()

FAQ #

When is my build script executed? #

If your project uses a build script, it is run implicitly in the following scenarios.

1. When poetry install is run, it is executed prior to installing the project’s root package.
2. When poetry build is run, it is executed prior to building distributions.
3. When a PEP 517 build is triggered from source or sdist by another build frontend.

How does Poetry ensure my build script’s dependencies are met? #

Prior to executing the build script, Poetry creates a temporary virtual environment with your project’s active Python version and then installs all dependencies specified under build-system.requires into this environment. It should be noted that no packages will be present in this environment at the time of creation.

Can I store the build script in a subdirectory? #

Yes you can. If storing the script in a subdirectory, your pyproject.toml might look something like this.

[tool.poetry]
...
packages = [
    { include = "package", from = "src"}
]
include = [
    { path = "src/package/**/*.so", format = "wheel" },
]
# exclude any intermediate source files
exclude = [
    "**/*.c"
]

[tool.poetry.build]
script = "scripts/build-extension.py"