Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Community Proposals
Community Proposals
tag:snake search within a tag
answers:0 unanswered questions
user:xxxx search by author id
score:0.5 posts with 0.5+ score
"snake oil" exact phrase
votes:4 posts with 4+ votes
created:<1w created < 1 week ago
post_type:xxxx type of post
Search help
Notifications
Mark all as read See all your notifications »
Q&A

Welcome to Software Development on Codidact!

Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.

Installing packages for, and using them with, a specific Python installation

+1
−0

I have multiple installations of Python on my machine. I already know how to choose one to use to run for my code - and therefore, I could also solve the problem for third-party application code.

However, almost all of the third-party code I work with is libraries, not applications. When I use Pip to install something from PyPI, I find that only one Python installation can use the code. Worse, the installation chosen by pip isn't necessarily the Python chosen by python.

Why don't (can't?) my Python installations share libraries? How do I choose one as the "target" for a Pip installation?

History
Why does this post require moderator attention?
You might want to add some details to your flag.
Why should this post be closed?

0 comment threads

1 answer

+1
−0

Choosing an environment for third-party libraries

The short version of the answer is: Python installations don't normally share libraries because there are too many ways it could go wrong; generally each installation has its own install locations for third-party libraries; and generally each Python installation also gets its own copy of Pip, which is designed to install for that Python installation.

Pip does offer some functionality for controlling the "target" install directory. However, for ordinary situations, simply use the copy of Pip that corresponds to the desired Python. This will be the simplest and most effective approach to the problem.

If you are using a virtual environment, make sure that it's properly activated and that the PATH hasn't been corrupted in some other way; and then pip should correspond to python automatically.

On Linux and Mac, this looks like:

python -m pip install package-installation-test

On Windows, you may prefer to allow the Python Launcher for Windows to choose Python for you:

py -m pip install package-installation-test

As long as python (or py) means the Python that you want to use the library with, this will install for the right Python (unless you have a serious system misconfiguration that needs to be diagnosed on a case-by-case basis).

If you need to select a different Python, do it the same way that you would when starting up Python normally: by explicitly providing the path to the Python executable, or by activating its virtual environment (if applicable).

Note: I am the author of the package-installation-test package on PyPI. I designed and created it in late 2022, specifically for the purpose of demonstrating the issue and allowing people to verify that the system works the way I describe it (and to test that they understand it).

Before getting into detail about how this system works, an important security note for Linux users especially:

Please don't try to solve the problem with sudo

Sometimes Linux users think they have found a workaround for this problem that involves sudo. However, this is dangerous and inappropriate. Some Linux distros don't even provide a working Pip for the built-in system Python, and block attempts to use Python's native functionality to obtain Pip. But even if you have a copy of Pip configured to install for the system Python, and wish to do so, you should do a user-level installation instead with the --user flag. (Recent versions of Pip should automatically assume --user when they discover that the expected target directory is write-protected.)

Python's "installation == environment" model

As noted above, normally each installation of Python has its own corresponding locations where third-party libraries can be installed, and its own corresponding copy of Pip.

In more detail: third-party libraries are typically installed to a site-packages subdirectory of the folder that contains the standard library. The documentation for the sysconfig standard library module gives more detail.

The underlying idea here is that each installation of Python corresponds to a separate Python environment. (If you need multiple environments that use the same version of Python, that's what virtual environments are for; they effectively simulate having a separate installation.)

Note that Pip itself is implemented in Python: it's a wrapper executable built around a pip package that is, itself, installed as a third-party library. How this gets installed is beyond the scope of this Q&A - but the point here is that, for the same reasons that Python installations can't share third-party libraries, they also can't share the Pip program. Since each one therefore needs to come with its own Pip, then, naturally each such Pip is configured to install for its own Python.

Thus, each environment naturally has its own copy of Pip, and using that Pip installs for its own environment.

Why installations can't "share" libraries generally

Although the Python development team tries very hard to ensure compatibility across Python versions, there are many reasons why this can't always happen:

  • The Python ABI, and the C API for Python, can change between Python minor versions. As a result, extension code written in C typically must be recompiled separately to work with each Python version (as well as being compiled for a specific hardware/OS combination).

  • Parts of the standard library are occasionally removed, which means some versions of certain libraries will stop working in newer versions of Python.

  • Code can be written that takes advantage of syntax that was added in a newer Python version; as a result, it won't work in an older version.

  • Code that was written for an older version of Python might not be syntactically valid in a new version. Of course, this was only deliberate in the case of the 2.x -> 3.x migration - but in rare cases, code could e.g. use a variable name that happens to become a keyword in a newer version.

How to force Python to try sharing libraries

As a result, it is not in general safe to try to use libraries that were specifically installed for one copy of Python, with another copy. While it's highly unlikely that any serious damage would occur this way, there is also a high chance that the code simply won't work, raising obscure SyntaxErrors or even segfaulting (if C extension code is involved).

However, if you know what you're doing and are willing to try it anyway, making Python use the installed libraries is a simple matter of ensuring that the containing site-packages folder is included in sys.path at the time of import. Note that this will give access to the entire suite of installed libraries. It will also probably not work to try to put a specific library's folder onto sys.path, because then the folder structure will be wrong for absolute imports.

To configure sys.path, we can do any of:

  • Use a .pth file, as described in the documentation for the site standard library module.

  • Set the PYTHONPATH environment variable to mention the desired site-packages folder.

  • Set the PYTHONHOME environment variable to override where Python looks for libraries by default.

This does not add a separate path to search for libraries, but instead replaces the default paths for libraries, including the standard library. The specified path should be the desired base standard library folder, not a site-packages sub-folder. This approach can only ever be expected to work for another environment with the same Python version - since no particular effort is made to ensure the standard library of one version would work with another.

  • At runtime, explicitly modify the value of sys.path. Modifying the existing list (e.g. by inserting a string) or replacing it with another list (assigning back to sys.path) will both work.

pip might not correspond to python - now what?

Sometimes, a system can be configured such that using the command pip doesn't find the "right" Pip executable - i.e., it ends up using (and installing for) a different Python than the one that the python command runs.

This problem seems to be more common on Windows, but can happen on any platform. Specific reasons will be explored in a different Q&A.

The important point for our purposes is that, again, Pip is implemented in Python, as a third-party library. Therefore, we can use the -m flag to ask Python to run the Pip package as a module. By doing this, we automatically control which Python installation is used - hence the solution at the top of this answer.

Virtual environments and other special cases

First: using the --user flag with Pip also allows installing to a user-specific directory - so that even if Python was installed with admin privileges, those privileges aren't required to use Pip; and so that different users on the same computer can set up their own separate Python environments without needing a separate Python installation. The necessary folder will be added, by default, to sys.path at Python startup, along with the usual site-packages folder.

However, the usual contemporary way to create a separate environment is to use a virtual environment, created with the standard library venv module. The "environment" that is created is a folder that works much like a separate Python installation - except that e.g. its "Python executable" might be a symlink to the "base" Python rather than a copy. In particular, it has its own site-packages subfolder; Pip is (by default) bootstrapped in; and there is a Pip executable in the same folder as the Python executable, even on Windows.

Therefore, as long as a virtual environment is active (using the activation script) - and as long as nothing else has re-modified the PATH environment variable after that - both python and pip commands will refer to that virtual environment. python is found in the virtual environment because the environment's path is added to the beginning of PATH; and so is pip; and the pip that was set up in that environment, installs for that environment. (This is yet another reason to use virtual environments even if you don't think you need them.)

History
Why does this post require moderator attention?
You might want to add some details to your flag.

0 comment threads

Sign up to answer this question »