Extending Python with Rust, Mojo, Cuda and C and building packages

Ruben Arts, Wolf Vollprecht

Friday 13:20 in Titanium3

Extending Python with native code is a common way of speeding up the execution. There are a number of traditional ways of writing Python extensions (Fortran, C, C++) but lately some modern languages have also entered the game (Rust, Mojo, and let’s count CUDA as modern, too).

All of these have slightly different ways of writing Python extensions, and they require the installation of a compiler and compilation tool chain, as well as possibly other system dependencies. Installing, updating and managing the system dependencies is usually a bit of a hassle, and it is where pixi comes in. Pixi is a new package manager that builds on top of the Conda ecosystem. The community distribution “conda-forge” already has tools like C and Rust compilers, and thus it’s easy to maintain the compiler + Python tool chain in a single project.

The new “pixi build” feature makes it even easier to build complex multi-language workspaces that combine different Python versions, compilers, and languages.

In the talk we will show lots of live demos going over simple numerical examples that highlight the different ways of extending Python (using pybind11, nanobind, PyO3, Mojo, …) glued together in a single workspace with pixi build compiling the extensions from source. We will also demonstrate how pixi can help not only depending on other packages from source, but also by building the packages into Conda and Wheel (PyPI) packages that can be shared.

After the talk, the listeners will have seen a number of ways how to extend Python code (easily!) with native languages and will have an understanding of benefits and drawbacks of the different approaches.

Ruben Arts

Wolf Vollprecht

Wolf Vollprecht has been active in the Python open source community for the past 5 years. He is a core member of conda-forge and the conda steering council, and the original author of the mamba package manager. He also has extensive experience in high-performance C++ and Rust. 2 years ago he started prefix.dev where the team is focusing all efforts on making cross-platform, language independent package management great (on top of the conda ecosystem).