Post History

## Choosing an environment for third-party libraries

The short version of the answer is: Python installations don't normally share libraries because there are too many ways it could go wrong; generally each installation has its own install locations for third-party libraries; and generally each Python installation *also gets its own copy of Pip*, which is designed to install for that Python installation.

<section class="notice is-success">

Pip does offer some functionality for controlling the "target" install directory. However, for ordinary situations, simply **use the copy of Pip that corresponds to the desired Python**. This will be the simplest and most effective approach to the problem.

If you are using a virtual environment, make sure that it's properly activated and that the `PATH` hasn't been corrupted in some other way; and then `pip` should correspond to `python` automatically.

On Linux and Mac, this looks like:

```bash
python -m pip install package-installation-test
```

On Windows, you may prefer to allow the Python Launcher for Windows to choose Python for you:

```cmd
py -m pip install package-installation-test
```

As long as `python` (or `py`) means the Python that you want to *use* the library with, this will *install* for the right Python (unless you have a serious system misconfiguration that needs to be diagnosed on a case-by-case basis).

If you need to select a different Python, do it the same way [that you would when starting up Python normally](https://software.codidact.com/posts/291616/291618#answer-291618): by explicitly providing the path to the Python executable, or by activating its virtual environment (if applicable).
</section>

**Note**: I am the author of the `package-installation-test` package on PyPI. I designed and created it in late 2022, specifically for the purpose of demonstrating the issue and allowing people to verify that the system works the way I describe it (and to test that they *understand* it).

Before getting into detail about how this system works, an important security note for Linux users especially:

### Please don't try to solve the problem with `sudo`

<section class="notice is-danger">

Sometimes Linux users think they have found a workaround for this problem that involves `sudo`. However, this is dangerous and inappropriate. Some Linux distros don't even provide a working Pip for the built-in system Python, and block attempts to use Python's native functionality to obtain Pip. But even if you have a copy of Pip configured to install for the system Python, and wish to do so, you should do a user-level installation instead with the `--user` flag. (Recent versions of Pip should automatically assume `--user` when they discover that the expected target directory is write-protected.)
</section>

## Python's "installation == environment" model

As noted above, normally each installation of Python has its own corresponding locations where third-party libraries can be installed, and its own corresponding copy of Pip.

In more detail: third-party libraries are *typically* installed to a `site-packages` subdirectory of the folder that contains the standard library. The [documentation for the `sysconfig` standard library module](https://docs.python.org/3/library/sysconfig.html#installation-paths) gives more detail.

The underlying idea here is that each installation of Python corresponds to a *separate Python environment*. (If you need multiple environments that use the same *version* of Python, that's what virtual environments are for; they effectively simulate having a separate installation.)

Note that Pip itself is implemented in Python: it's a wrapper executable built around a `pip` package that is, itself, installed as a third-party library. How this gets installed is beyond the scope of this Q&A - but the point here is that, for the same reasons that Python installations can't share third-party libraries, they also can't share the Pip program. Since each one therefore needs to come with its own Pip, then, naturally each such Pip is configured to install for its own Python.

Thus, each environment naturally has its own copy of Pip, and using that Pip installs for its own environment.

### Why installations can't "share" libraries generally

Although the Python development team tries very hard to ensure compatibility across Python versions, there are many reasons why this can't always happen:

* The Python ABI, and the C API for Python, can change between Python minor versions. As a result, extension code written in C typically must be recompiled separately to work with each Python version (as well as being compiled for a specific hardware/OS combination).

* Parts of the standard library are occasionally removed, which means some versions of certain libraries will stop working in newer versions of Python.

* Code can be written that takes advantage of syntax that was added in a newer Python version; as a result, it won't work in an older version.

* Code that was written for an older version of Python might not be syntactically valid in a new version. Of course, this was only *deliberate* in the case of the 2.x -> 3.x migration - but [in rare cases](https://stackoverflow.com/questions/51337939), code could e.g. use a variable name that happens to become a keyword in a newer version.

### How to force Python to try sharing libraries

<section class="notice is-danger">

**As a result, it is not in general safe to try to use libraries that were specifically installed for one copy of Python, with another copy.** While it's highly unlikely that any serious damage would occur this way, there is also a high chance that the code simply won't work, raising obscure `SyntaxError`s or even segfaulting (if C extension code is involved).

However, if you know what you're doing and are willing to try it anyway, making Python use the installed libraries is a simple matter of ensuring that the containing `site-packages` folder is included in `sys.path` at the time of import. Note that **this will give access to the *entire suite*** of installed libraries. It will also *probably not work* to try to put a specific library's folder onto `sys.path`, because then the folder structure will be wrong for absolute imports.
</section>

To configure `sys.path`, we can do any of:

* Use a `.pth` file, [as described](https://docs.python.org/3/library/site.html) in the documentation for the `site` standard library module.

* Set the `PYTHONPATH` environment variable to mention the desired `site-packages` folder.

* Set the `PYTHONHOME` environment variable to *override* where Python looks for libraries by default.

<section class="notice is-danger">

This *does not* add a separate path to search for libraries, but instead **replaces the default paths** for libraries, **including the standard library**. The specified path should be the desired base standard library folder, *not* a `site-packages` sub-folder. This approach **can only ever be expected to work** for another environment with the **same Python version** - since no particular effort is made to ensure the standard library of one version would work with another.
</section>

* At runtime, explicitly modify the value of `sys.path`. Modifying the existing list (e.g. by inserting a string) or replacing it with another list (assigning back to `sys.path`) will both work.

### `pip` might not correspond to `python` - now what?

Sometimes, a system can be configured such that using the command `pip` doesn't find the "right" Pip executable - i.e., it ends up using (and installing for) a different Python than the one that the `python` command runs.

This problem seems to be more common on Windows, but can happen on any platform. Specific reasons will be explored in a different Q&A.

The important point for our purposes is that, again, **Pip is implemented in Python**, as a third-party library. Therefore, we can use the `-m` flag to ask Python to run the Pip package as a module. By doing this, we automatically control which Python installation is used - hence the solution at the top of this answer.

### Virtual environments and other special cases

First: using the `--user` flag with Pip also allows installing to a user-specific directory - so that even if Python was installed with admin privileges, those privileges aren't required to use Pip; and so that different users on the same computer can set up their own separate Python environments without needing a separate Python installation. The necessary folder will be added, by default, to `sys.path` at Python startup, along with the usual `site-packages` folder.

However, the usual contemporary way to create a separate environment is to use a *virtual* environment, created with the standard library `venv` module. The "environment" that is created is a folder that works much like a separate Python installation - except that e.g. its "Python executable" might be a symlink to the "base" Python rather than a copy. In particular, it has its own `site-packages` subfolder; Pip is (by default) bootstrapped in; and there is a Pip executable **in the same folder as the Python executable, even on Windows**.

<section class="notice is-success">

Therefore, as long as a virtual environment is active (using the activation script) - and as long as nothing else has re-modified the `PATH` environment variable after that - both `python` and `pip` commands will refer to that virtual environment. `python` is found in the virtual environment because the environment's path is added to the beginning of `PATH`; and so is `pip`; and the `pip` that was set up in that environment, installs for that environment. (This is yet another reason to use virtual environments even if you don't think you need them.)
</section>