Managing Python Projects With pyproject.toml

pyproject.toml is a powerful configuration file format for Python projects that enhances project organization, build specifications, and dependency management.

It was introduced as part of PEP 518 and has become the standard for modern Python projects, earning its place as the default in major tools like Poetry, Hatch, and PDM.

With its broad ecosystem support and clean, readable syntax, pyproject.toml offers a vastly improved configuration experience compared to the legacy setup.py approach.

This article will guide you using pyproject.toml to create a well-structured Python project. You'll learn how to leverage its features and customize configurations to build a project that adheres to industry best practices.

Prerequisites

Before diving into the rest of this article, ensure you have a recent version of Python (3.13+) and pip installed on your machine. This article also assumes a basic understanding of Python and dependency management.

The history and purpose of pyproject.toml

For years, Python packaging was fragmented, relying on multiple configuration files like setup.py, requirements.txt, and tool-specific configs.

This led to inconsistencies, security risks, and maintenance challenges. In particular, the setup.py approach suffered from arbitrary code execution risks, bootstrapping issues, and lack of standardization.

To address these problems, PEP 518 introduced pyproject.toml in 2016, providing a standardized way to define build dependencies. Over time, additional PEPs refined its role, with significant tools like Poetry, Flit, and setuptools adopting it as a central configuration file.

Switching from setup.py to a declarative approach, pyproject.toml enhances security, simplifies dependency management, promotes consistency, and centralizes configuration for development tools.

Getting started with pyproject.toml

To get the most out of this tutorial, create a new Python project to try out the concepts we'll discuss.

Start by initializing a new project structure using the commands below:

mkdir pyproject-demo && cd pyproject-demo

Create the virtual environment:

python3 -m venv venv 

Activate the virtual environment:

source venv/bin/activate 

Now, create a basic pyproject.toml file in the root of your project directory:

[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "pyproject-demo"
version = "0.1.0"
description = "A sample Python project using pyproject.toml"
readme = "README.md"
authors = [
    {name = "Your Name", email = "your.email@example.com"}
]
requires-python = ">=3.7"

This snippet defines the build system requirements and basic project metadata. We'll explore all the different ways you can customize this configuration, but for now, let's set up a minimal package structure:

mkdir -p src/pyproject_demo

Create a simple module file:

"""A sample Python package using pyproject.toml."""

__version__ = "0.1.0"


def hello():
    """Return a friendly greeting."""
    return "Hello, world!"

Now, verify your setup by installing the package in development mode:

pip install -e .

Obtaining file:///Users/stanley/pyproject-demo
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
  ...
  Stored in directory: /private/var/folders/rr/372_1g9j1cbd1_zhrcc13s8m0000gn/T/pip-ephem-wheel-cache-25cy86ow/wheels/17/4b/2d/a93454a53e1a643831f489eb61407df39846d1d33d89c2428a
Successfully built pyproject-demo
Installing collected packages: pyproject-demo
Successfully installed pyproject-demo-0.1.0
...

Once installed, you can test the package in a Python interpreter. Open the shell with the following command:

python

Then, run the following code:

>>> from pyproject_demo import hello
>>> hello()
'Hello, world!'

The first thing you'll notice about pyproject.toml is that it uses TOML, a minimal configuration file format designed to be easy to read and write.

Unlike setup.py, which is executable Python code, pyproject.toml is a declarative specification, which eliminates many potential security issues and makes project configurations more consistent.

Understanding the core sections in pyproject.toml

Now that you've set up a basic project, let's explore the main sections of pyproject.toml in detail. Understanding these sections is crucial for effectively configuring your Python projects.

The build-system section

The [build-system] section is mandatory according to PEP 518. It specifies which build tools are required to build your package and which backend to use:

[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

The requires field lists the packages needed for building (not running) your package. The build-backend specifies which tool will interpret your build instructions. While setuptools is the most common backend, alternatives like Hatchling, Poetry, or Flit can also be used.

The project section

The [project] section contains metadata about your package and replaces the previous setuptools setup() call:

[project]
name = "pyproject-demo"
version = "0.1.0"
description = "A sample Python project using pyproject.toml"
readme = "README.md"
authors = [
    {name = "Your Name", email = "your.email@example.com"}
]
requires-python = ">=3.7"
license = {text = "MIT"}
classifiers = [
    "Development Status :: 3 - Alpha",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]

This section includes package name, version, description, and other metadata that will be used when publishing to the Python Package Index (PyPI). The requires-python field helps users know which Python versions are supported by your package.

Managing dependencies

One of the most significant improvements in pyproject.toml is how it handles dependencies. Dependencies are specified in the [project.dependencies] section:

[project]
# ... other project metadata ...

dependencies = [
    "requests>=2.28.0",
    "pyyaml>=6.0",
    "click>=8.1.0",
]

For optional features or development dependencies, you can use the [project.optional-dependencies] section:

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "black>=22.3.0",
    "flake8>=4.0.1",
    "mypy>=0.950",
]

docs = [
    "sphinx>=4.5.0",
    "sphinx-rtd-theme>=1.0.0",
]

This approach lets users install extra dependencies only when needed:

pip install -e ".[dev]"  # Install with development dependencies
pip install -e ".[docs]"  # Install with documentation dependencies
pip install -e ".[dev,docs]"  # Install with both sets

Entry points and scripts

To create command-line scripts that can be called directly after installation, use the [project.scripts] section:

[project.scripts]
pyproject-demo = "pyproject_demo.cli:main"

This configuration makes a command called pyproject-demo available on the system path after installation, which calls the main() function in the pyproject_demo.cli module.

For more complex plugin registration or entry points that other packages can discover, use the [project.entry-points] section:

[project.entry-points."console_scripts"]
pyproject-demo = "pyproject_demo.cli:main"

[project.entry-points."pytest11"]
pyproject-plugin = "pyproject_demo.pytest_plugin"

Tool-specific configuration in pyproject.toml

Beyond the standard project metadata and dependency management, one of the most powerful features of pyproject.toml is its ability to house configuration for various development tools. This approach centralizes your project settings, eliminating the need for multiple configuration files scattered throughout your project directory.

Black code formatter

Black is a popular Python code formatter that enforces a consistent style across your codebase. You can configure Black directly in your pyproject.toml file:

[tool.black]
line-length = 88
target-version = ["py37", "py38", "py39"]
include = '\.pyi?

exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

This configuration sets the maximum line length to 88 characters (Black's default), specifies which Python versions to target, and defines patterns for files to include or exclude from formatting. Black's native support for pyproject.toml makes it a perfect example of how modern Python tools are embracing this configuration approach.

MyPy type checking

MyPy is a static type checker for Python that helps catch type-related errors before runtime. Its configuration in pyproject.toml is straightforward:

[tool.mypy]
python_version = "3.7"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true

[[tool.mypy.overrides]]
module = ["tests.*"]
disallow_untyped_defs = false

This configuration enforces strict type checking for your project code while relaxing the requirements for test files. The disallow_untyped_defs option ensures all functions have type annotations, improving code quality and documentation. With the override section, you can apply different rules to specific modules, making it easier to adopt type checking in larger projects gradually.

Ruff

Ruff is a fast Python linter written in Rust that aims to replace multiple tools like Flake8, isort, and more. Its configuration in pyproject.toml is comprehensive:

[tool.ruff]
# Enable flake8-bugbear (B) rules
select = ["E", "F", "B"]
# Ignore specific rules
ignore = ["E501"]
# Line length to target
line-length = 88
# Target Python version
target-version = "py37"
# Allow autofix for all enabled rules
fixable = ["ALL"]
# Allow unused variables with leading underscore
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"

[tool.ruff.isort]
known-first-party = ["pyproject_demo"]

This configuration enables specific rule sets (E for style errors, F for flake8 rules, and B for bugbear rules), ignores certain rules (like E501 for line length), and configures import sorting behavior. Ruff's unified approach to linting demonstrates the value of having a single configuration file for all your development tools, as it can replace multiple separate linters with a single, faster implementation.

Combining multiple tools

The real power of pyproject.toml comes from centralizing all these configurations in one place. Here's how they look together:

[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

[project]
# ... project metadata ...

[tool.black]
line-length = 88
target-version = ["py37", "py38", "py39"]

[tool.mypy]
python_version = "3.7"
warn_return_any = true
disallow_untyped_defs = true

[tool.ruff]
select = ["E", "F", "B"]
line-length = 88
target-version = "py37"

This unified approach offers several advantages:

• Single source of truth: All configurations are in one file, making it easier to maintain consistency.
• Reduced cognitive load: Developers don't need to remember multiple file formats and locations.
• Easier onboarding: New team members can quickly understand the project's tooling setup.
• Version control efficiency: Changes to tool configurations are tracked together, simplifying code reviews.

Most modern Python development tools now support pyproject.toml configuration, reflecting a community-wide shift toward standardization and simplification.

As you develop your Python projects, leveraging this centralized configuration approach can significantly improve your development workflow and project maintainability.

Dynamic versioning with pyproject.toml

One common challenge in Python projects is maintaining consistent version numbers across your package. Repeating the version in multiple places can lead to inconsistencies when one location is updated but others are forgotten.

Let's explore how to implement dynamic versioning with pyproject.toml to solve this problem.

Using a dedicated version file

A good practice is to create a single source of truth for your version number:

"""Version information."""

__version__ = "0.1.0"

Then reference this version in your pyproject.toml file:

[project]
name = "pyproject-demo"
dynamic = ["version"]
# ... other project metadata ...

[tool.setuptools.dynamic]
version = {attr = "pyproject_demo._version.__version__"}

With this approach, you declare that the version is "dynamic" in the [project] section and specify its source in the [tool.setuptools.dynamic] section. This configuration tells setuptools to read the version from the __version__ variable in the pyproject_demo._version module at build time.

The primary benefit of this method is that both your code and build system reference the same version string. Your package can access its version at runtime using:

from pyproject_demo._version import __version__

print(f"Running version {__version__}")

Using SCM-based versioning

For more advanced projects, you can use tools like setuptools-scm, which automatically derives your package version from git tags:

[build-system]
requires = ["setuptools>=42", "wheel", "setuptools_scm[toml]>=6.2"]
build-backend = "setuptools.build_meta"

[project]
name = "pyproject-demo"
dynamic = ["version"]
# ... other project metadata ...

[tool.setuptools_scm]
write_to = "src/pyproject_demo/_version.py"

This configuration adds setuptools_scm to your build dependencies and tells it to write the version (derived from git tags and local changes) to a _version.py file.

When you tag a commit in your git repository (e.g., git tag -a v0.2.0 -m "Version 0.2.0"), setuptools-scm will use that tag as the version. For local development with uncommitted changes, it will add development suffixes (like .dev1+g4f8e9d1.d20230315), helping you track exactly which code version is being used.

The version file is generated during the build process, so it won't exist in the source repository but will be available in the installed package:

>>> import pyproject_demo
>>> pyproject_demo.__version__
'0.2.0'  # Or something like '0.2.0.dev1+g4f8e9d1.d20230315' for development builds

Manually managing versions in pyproject.toml

For simpler projects, you might prefer to maintain the version directly in pyproject.toml:

[project]
name = "pyproject-demo"
version = "0.1.0"
# ... other project metadata ...

However, you'll need a way for your package to access this version at runtime. One approach is to use the importlib.metadata module (available in Python 3.8+ or via the importlib-metadata backport):

"""A sample Python package using pyproject.toml."""

try:
    from importlib.metadata import version, PackageNotFoundError
except ImportError:
    from importlib_metadata import version, PackageNotFoundError

try:
    __version__ = version("pyproject-demo")
except PackageNotFoundError:
    # Package is not installed
    __version__ = "unknown"


def hello():
    """Return a friendly greeting."""
    return "Hello, world!"

This method works well for installed packages but has limitations during development. It's best suited for simple projects or when integration with development tools that expect a static version number is not needed.

Final thoughts

This article explored how pyproject.toml has transformed Python project management through centralized, declarative configuration.

The adoption of pyproject.toml by the Python ecosystem demonstrates a commitment to standardization that benefits developers with enhanced security, simplified workflows, and better dependency management.

Consider migrating your existing projects to pyproject.toml, starting simple and gradually incorporating more advanced features. Effective Python project management is an ongoing refinement process, and embracing this approach will serve you well as your projects evolve.

Happy coding!

Got an article suggestion? Let us know

Next article

A Comprehensive Guide to Profiling in Python

Discover how to quickly identify and resolve performance issues in your Python applications using Python's built-in profiling tools. In this step-by-step guide, you'll explore manual timing, profiling with `cProfile`, creating custom decorators, visualizing profiling data with SnakeViz, and applying practical optimization techniques.

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.