Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
Understanding "subprocess-exited-with-error" from Pip
This question is about the general case of subprocess-exited-with-error. It's intended to help you understand the situation if you get an error like this, and there are some common workarounds - but solving the problem directly will require more analysis.
Sometimes when I try to install a package from PyPI with Pip, I get a long error message, and the output ends like:
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
What exactly is happening, and why? In particular, what "subprocess" is Pip talking about? Can I do anything about it?
1 answer
Problems like this start when Pip retrieves your desired package as a sdist ("source distribution"), which has to be separately built on your machine before it can be installed. Normally, package authors pre-build the package for you so that Pip can get it directly as a wheel; but when the package includes code in other languages besides Python, this isn't always feasible. (Some Python-only packages simply haven't done this because they didn't know it was expected. Older packages could be from a time before there was an expectation, or even before there were wheels. But these are usually trivial to build.)
Because an sdist could contain pretty much anything, that might need to be built in all sorts of ways (for example, Numpy uses both C and Fortran code; many newer packages use languages like Rust or Go as well as C), Pip needs to follow instructions included with the package. There needs to be a starting point written in Python, but it can do anything from there (for example, using subprocess to run a C compiler). Pip will install any dependencies that the build script declares, call into the code using a well-defined API, and then use the resulting wheel to finish the installation.
Common workarounds
Of course, the above can go wrong in many ways. But regardless of the exact problem, here are some things you can try first:
Look for wheels for the project that failed to install
Go to the project page on PyPI for the package you want (this looks like https://pypi.org/project/<project_name>), and choose "Download files" to see what wheels are available. Don't download anything - if it were usable, Pip would have found it. But pay attention to whether there are any wheels listed, and to the filenames for the available wheels. These follow a strict structure; basically they let you know what versions of Python, what operating systems, and what architectures (AMD/Intel/legacy 32-bit) are explicitly supported with pre-built wheels.
If you don't see anything for the version of Python you're using, it could be that your version of Python isn't supported at all. But for new Python options, the developers just might not have tested it.
You can also check previous versions of the package (if you were trying to install a specific version, you can find it in the list, and also check other versions). Choose "Release history" in the sidebar, choose a version, and then choose "Download files" again.
Check the project issue tracker
A lot of the time, you won't be the first to have run into the problem. If you are, you can report that something happened and they can investigate.
Check the issue tracker for the build backend etc.
Hardly anyone wants to write code to do things like locating a usable C compiler on your system, or writing the little book-keeping files that the wheel standard requires, or putting everything into the right place for the archive (a wheel is basically a renamed zip file). In practice, they delegate to a build backend such as Setuptools instead. In the configuration, they direct Pip to ensure the backend is installed as a build-time dependency; Pip will do that, hook into the backend's code, and point it towards the unpacked sdist for the rest. The build backend might call additional code provided by the package (for Setuptools, this is in the setup.py file), or it might be able to figure everything out from configuration data.
If you can get access to the config information (for example, by browsing the repository on GitHub, or downloading the sdist from PyPI and opening it) and you see a pyproject.toml file, this can tell you what build backend is being used. (If you don't see a [project] section, or if there is no pyproject.toml at all, assume Setuptools.)
Wait
By default, Pip automatically uses build isolation. Every package it has to build from source gets a separate environment - so that building one package doesn't interfere with the next, and so that different packages can (if necessary) use different versions of the same build backend.
Usually, packages are compatible with a wide range of build backend versions. But Pip will grab the latest version of that dependency unless it's told otherwise, and packages usually don't say otherwise. When Setuptools introduces a breaking change, it can cause problems for a huge number of packages. On a few occasions, they've rolled back the change promptly and released a new version that Pip will use instead. They may even "yank" the offending version, so that it can only be installed by people who explicitly request it.
I'm writing this a few days after one such incident (the change was reverted in about 5 hours), but it's definitely not the first time this has happened. In theory, every major version of Setuptools could cause a breakage for somebody, but some changes are much more impactful than expected.
Set up a build environment manually
If you've determined that a new version of a build backend has caused the problem and you can't wait for a fix, you can try forcing an older version by setting up the build environment yourself. (If you want the builds to be "isolated", you'll need to make that environment fresh, install Pip in it, and do this separately for each package you want to install from source.)
The process looks like:
-
Explicitly install the build backend, constraining the dependency, e.g.:
pip install "setuptools<78"You need to quote the argument to avoid the special meaning of
>or<on the command line. -
Install using the
--no-build-isolationflag, e.g.:pip install --no-build-isolation --no-binary=:all: package-installation-test(I chose this package as a safe example. It provides a wheel, so
--no-binaryis used to force trying the sdist.)
Constrain the build isolation
Pip, and other installers, may give you a way to put additional constraints on dependencies on top of what the packages themselves demand. For Pip, you can make these apply to the build dependencies, too - but only using the PIP_CONSTRAINT environment variable, not with the --constraint option.
First, create a constraints file - the name doesn't matter, but we'll call it constraints.txt for this example. In the file, write:
setuptools < 78
Or whatever other constraint is necessary. Then, try the install but with the PIP_CONSTRAINT environment variable set to the constraint file's name, e.g:
PIP_CONSTRAINT=constraints.txt pip install --no-binary=:all: package-installation-test
(Again, --no-binary=:all: is only here for demonstration purposes).
On Windows there is no temporary setting for environment variables; you will need to SET PIP_CONSTRAINT=constraints.txt in a separate command, and possibly unset or restore it later.
0 comment threads
0 comment threads