Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
Why does `distutils` seem to be missing or broken? Isn't it part of the standard library?
Sometimes when I try to install a third-party library for Python, I get an error saying that either distutils, or some part of it like distutils.core or distutils.util, could not be found. It's shown as a ModuleNotFoundError, so presumably it's coming from Python when it tries to run a setup.py script for installation.
I've also seen similar messages when using libraries that were already installed. For example, if I try installing the popular SpeechRecognition third-party library in Python 3.12, I can reproduce this bug report:
Python 3.12.2 (main, Mar 29 2024, 00:26:08) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import speech_recognition as sr
>>> sr.Microphone()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/path/to/lib/python3.12/site-packages/speech_recognition/__init__.py", line 80, in __init__
self.pyaudio_module = self.get_pyaudio()
^^^^^^^^^^^^^^^^^^
File "/path/to/lib/python3.12/site-packages/speech_recognition/__init__.py", line 111, in get_pyaudio
from distutils.version import LooseVersion
ModuleNotFoundError: No module named 'distutils'
Why do problems like this occur? Isn't distutils part of the standard library?
I've heard some suggestions that installing setuptools can fix the problem. Why and how does this work?
1 answer
Understanding distutils and setuptools: the history
The short version of "why does installing Setuptools fix the problem?" is that Setuptools provides its own distutils package, overriding anything the standard library might or might not have. When you import setuptools explicitly, it furthermore ensures that a patched version of distutils is imported.
Way back in the prehistory of Python, the Distutils package was created so that there would be some standard way of distributing and installing each others' Python code - including compiling C extensions, with some nice wrappers to deal with other computers having different C compilers available. It was made available when Python 1.5 was current, and became part of the standard library in 1.6.
However, over time a lot of weaknesses were exposed in this system, and people also started to suspect that relying on the standard library in this particular way might be a mistake. Multiple third-party alternatives popped up, and Setuptools won out. To this day, Setuptools is still technically third-party.
For backwards-compatibility reasons, Setuptools worked by patching the standard library distutils package, in addition to providing its own modules.
Starting around Python 3.4, using Distutils directly became "informally deprecated"; the Python documentation started to recommend using Setuptools instead, and moved the actual information about how to use Distutils into a separate legacy documentation section. In Python 3.10 the deprecation became official; and following the usual deprecation policy, the distutils package was removed from the standard library in Python 3.12. Since Setuptools can no longer depend on the standard library including distutils at all, it now includes its own "vendored" version of the old distutils, and applies its own modifications on top of that.
Fixing problems with others' code, in Python 3.12
To fix problems running code that relies on distutils, install Setuptools in the environment where the code will run.
If you have a problem installing code on 3.12 because of distutils, the situation is a lot more complicated.
Usually there is no problem with installing a library that relies on distutils. Ideally, the library was pre-built into a wheel, so all the parts that rely on distutils have been done ahead of time. If that didn't happen (which means Pip will try to install from source, unless told to give up), the library is supposed to be responsible for specifying build requirements. Newer projects that use a pyproject.toml file can explicitly say that the build environment should contain setuptools, and that will make Pip install Setuptools first. When the pyproject.toml doesn't say what "build system" to use, Setuptools is supposed to be the default anyway.
However, in Python 3.12, circumstances can combine to create a very annoying problem. Pip tries to use build isolation when it installs from source, meaning that it creates a separate virtual environment for the installation process (so that, for example, the setup process can use specific versions of Setuptools and other such tools, even if the code will be installed into an environment with a different version of those tools). Starting with Python 3.12, virtual environments created using the venv standard library (like how Pip does it) don't include Setuptools by default. This sometimes causes installations to fail: even though Setuptools is installed in the Python where we want to install the new library, it isn't installed in the Python where it will be built.
In these cases, you can try passing the --no-build-isolation flag to Pip, to make it use the Setuptools (and potentially any other previously installed libraries!) from the existing environment while trying to install the new library. If that doesn't work, the only remaining option is to file a bug for that library (or check if the author has already said anything about it).
Fixing problems with others' code, in earlier Python versions
If distutils.core, distutils.util etc. seem to be missing on your system, especially on Linux, this is probably because you are using a copy of Python that came with the system that has these components disabled. Your best bet is to use a virtual environment and make sure Setuptools is installed in that virtual environment (this should happen automatically by default). Otherwise, you can check with your distro to see if there is a system-level package you can install to add Distutils to your system Python.
Be aware of the usual risks involved in modifying a program that came with your operating system - especially since this is taking a step towards making it possible to add other third-party code to your system Python directly, bypassing the system package manager. (For the same reason, your system Python might not include Pip.) If someone managed to get malware onto PyPI and you inadvertently installed it this way, it could result in serious damage. Even ordinary third-party packages could potentially interfere with critical system Python scripts.
Fixing problems with your own code
If you must maintain code that depends on distutils functionality and want it to work in Python 3.12 (especially if you are distributing it to others), consider replacing uses of distutils that can be covered by other standard library modules. If you still have parts for which "setuptools is the best substitute":
-
Install Setuptools locally
-
If your code uses the functionality at runtime, make sure that Setuptools is listed as a runtime dependency - typically, by mentioning it in the
project.dependenciesinpyproject.toml. -
If at all possible, build wheels locally and distribute them as well, even if your project uses only Python source code.
-
If you must distribute source that must use Distutils during the installation process, make sure your project has a
pyproject.toml(add one, if it didn't already) that explicitly lists Setuptools as a build-time dependency, by describing it in the[build-tool]table. If applicable, modify yoursetup.pyso thatsetupgets imported from Setuptools rather than Distutils (i.e., replacefrom distutils.core import setupwithfrom setuptools import setup).
0 comment threads
0 comment threads