Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
Why does Pip display "error: externally-managed-environment", and what can I do about it?
My (non-Windows) operating system came with Python, but that Python didn't include Pip. I followed instructions to install Pip for the included Python, using my system's package manager.
But now when I try to use Pip, I[1] get errors like:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
What's going on here? What does it mean by "externally managed", and why should I have to jump through the hoops described?
-
In reality, my system doesn't actually work this way - and I haven't actually installed Pip for my system Python, because I personally recommend against doing so. The error message shown here is copied from the corresponding Stack Overflow question (alternately). ↩︎
2 answers
The following users marked this post as Works for me:
| User | Comment | Date |
|---|---|---|
| Karl Knechtel | (no comment) | Jun 27, 2024 at 19:05 |
Reminder: never use sudo to run Pip. This can never properly fix a problem and only introduces serious security risks. Installing third-party packages from PyPI can run arbitrary code at install time, by design, so it should never be given root privileges.
Anyone who has observed using sudo for Pip to solve a problem, was misled. The only reason it might have helped is because the root user has a different PATH, resulting in
a different Pip being used, corresponding to a different Python installation.
(Example commands involving sudo are deliberately not shown here, to avoid mindless copy-pasting.)
Overview
The situation is exactly as the error message describes, and the advice given is useful and will solve the problem:
-
Your Python environment (i.e., the Python that was installed with your system, plus the libraries etc. that come with it) is "externally managed".
-
Because of this, you are expected not to use Pip to install packages in that environment, but instead use the system package manager.
If the system package manager doesn't provide the third-party library that you want - or if you just don't want to use the system package manager - the most logical alternative is to set up a separate Python environment, by using a virtual environment. This is the overall simplest and safest approach - leave the Python that came with the system alone, as much as possible, and use a separate Python for development.
"External management" and PEP 668
When Pip calls the environment "externally managed", it means that some other program or tool is supposed to be responsible for installing packages, instead of Pip. Since Python 3.11 and Pip 23.0, Pip has implemented the "marker" standard described in PEP 668 (the documentation for this feature is now maintained separately).
This allows Linux distro maintainers (and others) to add a specially named "marker" file to the folder that contains Python's standard library. When Pip sees that there is such a file and sees that it is not using a virtual environment, the error occurs. (Pip needs to disable the check for virtual environments because they use the base Python's standard library.)
The marker file can optionally include specific advice from the maintainer, which Pip will include in the error. The indented part after the arrow comes from the file; the error itself, as well as the "note" and "hint" at the bottom, are hard-coded by Pip.
Regarding the Debian-specific advice shown here: python3-full is a dependency package which will add multiple pieces to the system Python which are disabled by default - including venv (which you'll need in order to make virtual environments), Tkinter (and IDLE), distutils (but not if your system Python is 3.12), and the "2to3" tool. It also adds a support library for GNU dbm database files. (However, it does not appear to include Pip.)
apt install python3-venv is enough if you just want to set up virtual environments. The base Python might have distutils anyway; the 2to3 tool is irrelevant for almost everyone now; dbm files are a niche interest; and new virtual environments will bootstrap Pip. However, your virtual environments won't include Tkinter unless they're created from a base that does. Installing python3-full may be worthwhile if you need both venv support and Tkinter support.
Of course, it's also possible to just do a completely separate Python installation. This would also allow you to choose a different version of Python.
Note that the advice about pipx is about installing Python applications, not libraries. As such, it's not really relevant for developers, usually.
Other workarounds
It's possible to circumvent the marker in a variety of ways, but please keep in mind that it exists for your protection. Note that even when circumvented, Pip will default to a user-level installation (as if you used the -U/--user option) if it doesn't have root privileges. Again, never use sudo to run Pip.
Keep in mind that even user-level installs can still "break" system packages by, for example, shadowing the names of standard library or other modules that the system-included scripts would otherwise import.
Options here include:
-
Pass the
--break-system-packagesoption to Pip, as described. This is a simple confirmation that you accept the risk. -
Set the
PIP_BREAK_SYSTEM_PACKAGESenvironment variable (also see official documentation), or specify thebreak-system-packagesoption in your Pip configuration.To change the Pip configuration, edit your
~/.config/pip/pip.conffile to include:[global] break-system-packages = trueor use the
pip configcommand:python -m pip config --global set global.break-system-packages trueThese approaches tell Pip to act as if the
--break-system-packagesflag were present for every command used. -
Rename or remove the marker file, which is named
EXTERNALLY-MANAGEDand will generally be in a location like/usr/lib/python3.x(according to the Python version). This will generally require root privileges.
0 comment threads
tl;dr: There's a few ways to bypass this:
- For all users on the machine: Get rid of
/usr/lib/python3.foo/EXTERNALLY-MANAGED. To prevent your package manager from adding it back, replace it with a dummy/empty file. (you'll have to do this again for every minor Python version) - For your user only: Create or edit
~/.config/pip/pip.confso that it contains:[global] break-system-packages = true - For one shell session: Set the environment variable
PIP_BREAK_SYSTEM_PACKAGES=1(you can configure your shell to always set this) - For one command only: Pass
--break-system-packagesto pip.
(there used to be some thoughts here on why this has become necessary in recent years, but it proved controversial and I removed it)
0 comment threads