I recently built a Python library for the first time in a while, and to make it available in PyPI I followed the official Python packaging guide. But it turns out there’s some rough edges in the suggested tools.
Namely, the build backend hatch puts all the files in the project directory into the built sdist, except from some files it chooses to exclude, which led to a gitignored file as well as VCS metadata getting packaged and uploaded to PyPI. This really surprised me, and I think most Python users new to packaging would be better served by choosing a conservative build backend like uv_build, which puts only the Python source modules and some specific project-level files into the sdist.
Outline:
- Following the official guide can bite you
- A nearly minimal example Python packge with
jj - Building the example package and finding it contains things it shouldn’t
- hatch wants to upload every file in your source tree
- The fix: use a conservative build backend like
uv_build - Conclusion: newbies should avoid hatch
Following the official guide can bite you
If you follow the official guide, it will point you at the official build
frontend tool called “build” and an official build
backend called hatch. Once configuring hatch in
pyproject.toml you don’t have to worry about it too much; running
pyproject-build will invoke hatch for you to build your package, ready to
upload to PyPI with something like twine.
A nearly minimal example Python packge with jj
Let’s try it out, making a simple project built using the recommended source tree layout:
packaging_tutorial/
├── LICENSE
├── pyproject.toml
├── README.md
├── src/
│ └── example_package_YOUR_USERNAME_HERE/
│ ├── __init__.py
│ └── example.py
└── tests/
I’m not quite sure why their example package ends with _YOUR_USERNAME_HERE;
I’m assuming this is a typo. Since I might want future collaborators to work on
the project, putting my username here seems odd, and I’ll leave it out. I don’t
think this will cause an issue. Anyway, let’s create the Python package:
$ mkdir -p src/example
$ cat > src/example/__init__.py
if __name__ == '__main__':
print('Hello world!')
Then fill in pyproject.toml with the suggested contents
$ cat > pyproject.toml
[build-system]
requires = ["hatchling >= 1.26"]
build-backend = "hatchling.build"
[project]
name = "example"
version = "0.1.0"
Let’s add a .gitignore so that any package builds don’t get added to source
control:
$ cat > .gitignore
dist/
*.pyc
I’ll add my PyPI API token to a .env file in here so I can easily publish my
package when I’m ready. This isn’t a great practice, but it’s only for a simple
local example, and the file is ignored thanks to my user-level gitignore
configuration, so while I don’t condone doing this, it shouldn’t pose a problem
for this simple trial.
$ cat > .env
PYPI_TOKEN=mysecret
I’m a fan of the Jujutsu VCS so let me use that to create a repo and a first commit:
$ jj git init --colocate
$ jj status
Working copy changes:
A .gitignore
A pyproject.toml
A src/example/__init__.py
Working copy (@) : nnntvyzv deead7e0 (no description set)
Parent commit (@-): zzzzzzzz 00000000 (empty) (no description set)
$ jj commit -m "initial commit"
Working copy (@) now at: muyynyzk 99d1736d (empty) (no description set)
Parent commit (@-) : nnntvyzv 3dda6c67 initial commit
Everything’s all set. Now, let’s build the package.
Building the example package and finding it contains things it shouldn’t
Using the PyPA build tool is pretty straightforward:
$ pyproject-build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- hatchling >= 1.26
* Getting build dependencies for sdist...
* Installed build dependency versions:
- hatchling==1.32.3
* Building sdist...
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- hatchling >= 1.26
* Getting build dependencies for wheel...
* Installed build dependency versions:
- hatchling==1.32.3
* Building wheel...
Successfully built example-0.1.0.tar.gz and example-0.1.0-py2.py3-none-any.whl
Great! Now let’s double check what’s in there before we upload…
$ tar -tzf dist/example-0.1.0.tar.gz
example-0.1.0/.env
example-0.1.0/.jj/.gitignore
example-0.1.0/.jj/repo/index/type
[ ... tons more .jj/ files cut for brevity ... ]
example-0.1.0/src/example/__init__.py
example-0.1.0/pyproject.toml
example-0.1.0/PKG-INFO
… wait, what??? Why does the sdist include .jj/ as well as my ignored .env
file???
hatch wants to upload every file in your source tree
Surprisingly, hatch doesn’t default to just packaging the Python code in your
src/ directory. Instead, it will grab every file in the directory tree
except for those satisfying pre-defined exclusion rules, like matching a name
in an internal constants
module,
and files matched by .gitignore. But… not all of your ignored files will
be excluded, either.
It directly parses .gitignore
See this code in backend/src/hatchling/builders/config.py on lines 751-763:
@cached_property
def vcs_exclusion_files(self) -> dict[str, list[str]]:
exclusion_files: dict[str, list[str]] = {"git": [], "hg": []}
local_gitignore = locate_file(self.root, ".gitignore", boundary=".git")
if local_gitignore is not None:
exclusion_files["git"].append(local_gitignore)
local_hgignore = locate_file(self.root, ".hgignore", boundary=".hg")
if local_hgignore is not None:
exclusion_files["hg"].append(local_hgignore)
return exclusion_files
hatch has hardcoded awareness of Git and Mercurial and specifically searches for
project-level .gitignore and .hgignore to read those files directly. But it
doesn’t handle user-level or system-level config, so anything you have ignored
from config outside of your project, like .env in my example, will not be
ignored by hatch, and will wind up in your sdist.
The fix: use a conservative build backend like uv_build
Changing the build backend to one that’s more intentional when choosing which
files to package will fix this problem
entirely. uv_build is
one such backend. In pyproject.toml, change the [build-system] section to
this:
[build-system]
requires = ["uv_build>=0.12.17,<0.13"]
build-backend = "uv_build"
Now rebuild the sdist:
$ rm dist/*
$ pyproject-build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- uv_build>=0.12.17,<0.13
* Getting build dependencies for sdist...
* Installed build dependency versions:
- uv-build==0.12.17
* Building sdist...
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- uv_build>=0.12.17,<0.13
* Getting build dependencies for wheel...
* Installed build dependency versions:
- uv-build==0.12.17
* Building wheel...
Finally, let’s check its contents:
$ tar -tzf dist/example-0.1.0.tar.gz
example-0.1.0/PKG-INFO
example-0.1.0/pyproject.toml
example-0.1.0/pyproject.toml.orig
example-0.1.0/
example-0.1.0/src
example-0.1.0/src/example
example-0.1.0/src/example/__init__.py
Great! Much more sensible. Rather than following heuristics and exclusion rules,
uv_build grabs only Python source files and pyproject.toml (as well as other
common relevant project files you might have, like LICENSE).
Conclusion: newbies should avoid hatch
If you know hatch well and know how to configure it, you can get it to mirror the behavior of uv_build. But by default, without configuration, hatch is very likely to package more than you expect into your sdist, which will get uploaded to PyPI or whatever other repository you choose. For users starting out, especially those using an alternative VCS like Jujutsu, Pijul, or Fossil, I’d suggest skipping hatch and just using uv_build. It has more reasonable and conservative defaults and is less likely to send files you don’t intend to PyPI for public consumption.