Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
Installing packages for, and using them with, a specific Python installation
I have multiple installations of Python on my machine. I already know how to choose one to use to run for my code - and therefore, I could also solve the problem for third-party application code.
However, almost all of the third-party code I work with is libraries, not applications. When I use Pip to install something from PyPI, I find that only one Python installation can use the code. Worse, the installation chosen by pip isn't necessarily the Python chosen by python.
Why don't (can't?) my Python installations share libraries? How do I choose one as the "target" for a Pip installation?
1 answer
The following users marked this post as Works for me:
| User | Comment | Date |
|---|---|---|
| Karl Knechtel | (no comment) | Jun 14, 2024 at 12:43 |
Choosing an environment for third-party libraries
The short version of the answer is: Python installations don't normally share libraries because there are too many ways it could go wrong; generally each installation has its own install locations for third-party libraries; and generally each Python installation also gets its own copy of Pip, which is designed to install for that Python installation.
Pip does offer some functionality for controlling the "target" install directory. However, for ordinary situations, simply use the copy of Pip that corresponds to the desired Python. This will be the simplest and most effective approach to the problem.
If you are using a virtual environment, make sure that it's properly activated and that the PATH hasn't been corrupted in some other way; and then pip should correspond to python automatically.
On Linux and Mac, this looks like:
python -m pip install package-installation-test
On Windows, you may prefer to allow the Python Launcher for Windows to choose Python for you:
py -m pip install package-installation-test
As long as python (or py) means the Python that you want to use the library with, this will install for the right Python (unless you have a serious system misconfiguration that needs to be diagnosed on a case-by-case basis).
If you need to select a different Python, do it the same way that you would when starting up Python normally: by explicitly providing the path to the Python executable, or by activating its virtual environment (if applicable).
Note: I am the author of the package-installation-test package on PyPI. I designed and created it in late 2022, specifically for the purpose of demonstrating the issue and allowing people to verify that the system works the way I describe it (and to test that they understand it).
Before getting into detail about how this system works, an important security note for Linux users especially:
Please don't try to solve the problem with sudo
Sometimes Linux users think they have found a workaround for this problem that involves sudo. However, this is dangerous and inappropriate. Some Linux distros don't even provide a working Pip for the built-in system Python, and block attempts to use Python's native functionality to obtain Pip. But even if you have a copy of Pip configured to install for the system Python, and wish to do so, you should do a user-level installation instead with the --user flag. (Recent versions of Pip should automatically assume --user when they discover that the expected target directory is write-protected.)
Python's "installation == environment" model
As noted above, normally each installation of Python has its own corresponding locations where third-party libraries can be installed, and its own corresponding copy of Pip.
In more detail: third-party libraries are typically installed to a site-packages subdirectory of the folder that contains the standard library. The documentation for the sysconfig standard library module gives more detail.
The underlying idea here is that each installation of Python corresponds to a separate Python environment. (If you need multiple environments that use the same version of Python, that's what virtual environments are for; they effectively simulate having a separate installation.)
Note that Pip itself is implemented in Python: it's a wrapper executable built around a pip package that is, itself, installed as a third-party library. How this gets installed is beyond the scope of this Q&A - but the point here is that, for the same reasons that Python installations can't share third-party libraries, they also can't share the Pip program. Since each one therefore needs to come with its own Pip, then, naturally each such Pip is configured to install for its own Python.
Thus, each environment naturally has its own copy of Pip, and using that Pip installs for its own environment.
Why installations can't "share" libraries generally
Although the Python development team tries very hard to ensure compatibility across Python versions, there are many reasons why this can't always happen:
-
The Python ABI, and the C API for Python, can change between Python minor versions. As a result, extension code written in C typically must be recompiled separately to work with each Python version (as well as being compiled for a specific hardware/OS combination).
-
Parts of the standard library are occasionally removed, which means some versions of certain libraries will stop working in newer versions of Python.
-
Code can be written that takes advantage of syntax that was added in a newer Python version; as a result, it won't work in an older version.
-
Code that was written for an older version of Python might not be syntactically valid in a new version. Of course, this was only deliberate in the case of the 2.x -> 3.x migration - but in rare cases, code could e.g. use a variable name that happens to become a keyword in a newer version.
How to force Python to try sharing libraries
As a result, it is not in general safe to try to use libraries that were specifically installed for one copy of Python, with another copy. While it's highly unlikely that any serious damage would occur this way, there is also a high chance that the code simply won't work, raising obscure SyntaxErrors or even segfaulting (if C extension code is involved).
However, if you know what you're doing and are willing to try it anyway, making Python use the installed libraries is a simple matter of ensuring that the containing site-packages folder is included in sys.path at the time of import. Note that this will give access to the entire suite of installed libraries. It will also probably not work to try to put a specific library's folder onto sys.path, because then the folder structure will be wrong for absolute imports.
To configure sys.path, we can do any of:
-
Use a
.pthfile, as described in the documentation for thesitestandard library module. -
Set the
PYTHONPATHenvironment variable to mention the desiredsite-packagesfolder. -
Set the
PYTHONHOMEenvironment variable to override where Python looks for libraries by default.
This does not add a separate path to search for libraries, but instead replaces the default paths for libraries, including the standard library. The specified path should be the desired base standard library folder, not a site-packages sub-folder. This approach can only ever be expected to work for another environment with the same Python version - since no particular effort is made to ensure the standard library of one version would work with another.
- At runtime, explicitly modify the value of
sys.path. Modifying the existing list (e.g. by inserting a string) or replacing it with another list (assigning back tosys.path) will both work.
pip might not correspond to python - now what?
Sometimes, a system can be configured such that using the command pip doesn't find the "right" Pip executable - i.e., it ends up using (and installing for) a different Python than the one that the python command runs.
This problem seems to be more common on Windows, but can happen on any platform. Specific reasons will be explored in a different Q&A.
The important point for our purposes is that, again, Pip is implemented in Python, as a third-party library. Therefore, we can use the -m flag to ask Python to run the Pip package as a module. By doing this, we automatically control which Python installation is used - hence the solution at the top of this answer.
Virtual environments and other special cases
First: using the --user flag with Pip also allows installing to a user-specific directory - so that even if Python was installed with admin privileges, those privileges aren't required to use Pip; and so that different users on the same computer can set up their own separate Python environments without needing a separate Python installation. The necessary folder will be added, by default, to sys.path at Python startup, along with the usual site-packages folder.
However, the usual contemporary way to create a separate environment is to use a virtual environment, created with the standard library venv module.
As long as a virtual environment is active (using the activation script) - and as long as nothing else has re-modified the PATH environment variable after that - both python and pip commands will refer to that virtual environment. (This is yet another reason to use virtual environments even if you don't think you need them.) See the linked Q&A for details on how this works.
0 comment threads
0 comment threads