Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
How to troubleshoot ModuleNotFoundError?
I ran some Python code and it crashed with ModuleNotFoundError. What is the general approach for dealing with these situations?
2 answers
Root cause
ModuleNotFoundError is a kind of ImportError. It occurs when Python code tries to use an absolute import, but can't find the named module - just as the name suggests. (Failed relative imports will only raise the base ImportError. This makes sense because an absolute import involves searching for the module, but for a relative import, Python should already know exactly where to look.)
Understanding sys.path
In general, Python looks for absolute imports by searching within folders that are named in the path attribute of the sys standard library module (sys.path). User code can modify or reassign the list like any other list; and reassigning it does affect module lookup.
However, this is rarely necessary and should always be carefully considered. Popular, complex third-party libraries often span tens of thousands, or even hundreds of thousands of lines of code without needing to do this.
At startup, sys.path is initialized according to clearly documented rules. Rather than trying to modify sys.path from within the code before attempting an import, it's normally a better idea to set the PYTHONPATH environment variable.
Normally, when Python is asked to do an absolute import, it will check each folder listed in sys.path - the folder itself only, not searching in any subdirectories - for a .py file with the right name, or some other implementation (e.g. a compiled .pyc bytecode file contained within a __pycache__ subdirectory; in Python 2.x, the .pyc files were expected to be side-by-side with the corresponding .py source code). It's possible to modify this search process, but that's beyond the scope of this answer. Some "built-in" standard library modules, such as sys, bypass this search process; the module is directly created within the Python implementation. (Among other things, this makes it possible to import sys without needing sys.path to exist already.)
(TODO: factor out a separate Q&A for this?)
Troubleshooting
Always check for typos first, of course.
If you didn't write the code to be imported yourself, it's important to understand whether that code is part of the standard library (something that comes with Python) or is third-party (needs to be installed separately). Check the Python standard library documentation to know what is included.
Everything else must be installed separately. Normally, third-party code is installed using Pip (the pip command-line program) - but if you set up the Python environment using Anaconda, you should use its tool (conda) instead. Some packages have custom installation instructions; in these cases you will have to read their documentation first.
When importing from the standard library
If you are trying to import a standard library module, outside of a very few special cases, the module can really only be not found due to a typo. (Of course, other things can indeed go wrong.)
But, for completeness:
TKinter
This is part of the standard library, but in some cases it may not be included with a Python distribution. This especially happens on Linux and MacOS machines, for a copy of Python that came with the operating system. To get TKinter, you may need to use the system package manager (not Pip) to install it. However, it's often better to install a separate Python version. Every Python installer from https://www.python.org should include Tkinter (and configure everything else needed for it on your system) automatically, and the defaults for compiling from source on Linux should include it as well.
(TODO: separate Q&A for details on this problem)
zlib
Similarly, Python may be built in such a way that it doesn't include the zlib standard library module.
OS-specific modules
Certain modules may only be available depending on the operating system, because they are used to provide OS-level functionality. These are listed in the "MS Windows Specific Services" and "Unix Specific Services" sections of the standard library documentation.
Modules removed or renamed in Python 3
Several standard library modules were renamed or reorganized for Python 3. Some old guides may refer to the Python 2 names of standard library modules, or old code may need to use something else in order to update for Python 3.
Here is a comprehensive listing of the changes:
| Module removed or renamed | Workarounds |
|---|---|
_winreg |
renamed to winreg
|
ConfigParser |
renamed to configparser
|
copy_reg |
renamed to copyreg
|
Queue |
renamed to queue
|
SocketServer |
renamed to socketserver
|
markupbase |
renamed to _markupbase - n.b. this is not intended for public use; it's an implementation detail for html.parser
|
repr |
renamed to reprlib
|
test.test_support |
renamed to test.support
|
Tkinter (and several others) |
Use tkinter; the module was renamed to conform to standard naming conventions. There used to be many Tkinter "package contents" that were actually top-level modules; now tkinter is a proper package. |
anydbm, dbhash, dumbdbm, gdbm, whichdb
|
Use the dbm package. These used to be separate modules (including dbm itself); now dbm is a package including all this functionality. |
HTMLParser |
Use html.parser; see the new html package
|
htmlentitydefs |
Use html.entities
|
urllib2 |
Use urllib.error and urllib.request; see the urllib package. Python 2 had separate urllib and urllib2 modules; now urllib is a package that includes urllib2 functionality (and more). |
urlparse |
Use urllib.parse
|
robotparse |
Use urllib.robotparser
|
httplib |
Use http.client; see the http package
|
BaseHTTPServer, CGIHTTPServer, SimpleHTTPServer
|
Use http.server
|
Cookie |
Use http.cookies
|
cookielib |
Use http.cookiejar
|
xmlrpclib |
Use xmlrpc.client; see the xmlrpc package
|
DocXMLRPCServer, SimpleXMLRPCServer
|
Use xmlrpc.server
|
sets |
Removed; use the built-in set type instead. (This is extremely obscure; this library was added in 2.3, and the built-in set type was added in 2.4.) |
new |
Removed; use types instead. The API is different, so code will need to be changed. |
In most cases, the 2to3 tool should be able to fix these imports automatically.
Modules removed in Python 3.12
The What’s New In Python 3.12 document mentions that a few standard library modules are no longer available:
| Module removed | Workarounds |
|---|---|
asynchat |
Use the new asyncio module instead; the API is different, so code will need to be adapted. |
asyncore |
Use the new asyncio module instead; the API is different, so code will need to be adapted. |
distutils |
Upgrade to use Setuptools instead; but keep in mind that it's encouraged to use pyproject.toml as much as possible for configuration. Running the setup.py script directly is now deprecated; instead, Setuptools will invoke it for you during the building process. |
imp |
Use the new importlib module instead; the API is different, so code will need to be adapted. Some basic guidance is included in the documentation. |
smtpd |
The documentation recommends the third-party aiosmtpd package instead. |
Many more old modules will be removed in 3.13, following PEP 594.
When importing from a third-party package
As well as checking for typos, make sure that you know what name to import. If you installed a third-party library from PyPI, the name that should be used with the import statement in your code does not necessarily have anything to do with the name used to find the package on PyPI. The PyPI name doesn't even have to be a valid Python identifier (in particular, it can use - characters).
Here are some commonly encountered mismatches:
| PyPI package name | Package names to use in code |
|---|---|
pillow |
PIL (from PIL import Image etc.) |
python-dateutil |
dateutil |
beautifulsoup4 |
bs4 (from bs4 import BeautifulSoup) |
opencv-python |
cv2 |
google-api-python-client |
googleapiclient(e.g. from googleapiclient.discovery import build). Some older versions may use apiclient instead. |
python-docx(There is also a docx PyPI package, but it is abandoned and does not support 3.x) |
docx |
pywin32(previously pypiwin32 and probably other names) |
win32api, win32com etc. (many other top-level packages) |
pyserial |
serial |
If the code has the correct name and you have verified that the package has been installed (for example by using pip list, pip show with the PyPI package name, or trying to install again and getting a Requirement already satisfied message), this happens because the script is using a different Python environment rather than the one where the package was installed. (TODO: separate Q&A for this issue)
When importing your own code
If you have code from other projects that you wrote yourself and need to use in the current project, generally it will be better to make an installable package and install it locally - even if you don't want to distribute it on PyPI (yet?).
Failing that, however, you will need to make sure that sys.path includes the appropriate path for the other code.
If, on the other hand, you are trying to import code that is part of the same project, this should mostly be done using relative imports instead.
The recommended way to organize a Python project is to have most of the code inside a package folder (or sub-folders), perhaps with some simple "driver" scripts just outside. In normal cases, the search path will include the folder with the "driver" scripts, so they can easily do an absolute import of some specific module inside the package. Then, within the package, use exclusively relative imports to connect different parts of the project. This minimizes the dependence on the module search path and helps prepare for packaging the code properly for distribution.
That said, absolute imports usually don't cause a ModuleNotFoundError, because the default sys.path (see the previous section on sys.path) should make it possible to find the other code. The most common problem occurs when trying to use the code in interactive mode (from the Python interpreter prompt). In this case, sys.path will normally include a '' entry, which is a relative path to the current working directory. As a result, using code like os.chdir can prevent modules from being found in the expected place.
0 comment threads
ModuleNotFoundError means that in running your program, Python attempted to import some module xyz but it was not present on your system.
It could be that you forgot to install the module (forgot to pip install -r requirements.txt), misspelled the module name, or tried to import a module that doesn't exist. But that would be somewhat obvious, so let's assume it's not that, and the module name is correct. However, do note that some packages on PyPI for example have surprising import names - sometimes you pip install xyz but import pyxyz etc - this should normally be covered in the package's documentation.
The first useful thing is to find a way to test the import easily. If you got ModuleNotFoundError: No module named 'xyz', you can reproduce it by running python -c 'import xyz' in your shell: If the import succeeds, it will print nothing and exit with code 0. If the import fails, you will see the ModuleNotFoundError again.
Often, package imports fail due to an environment mismatch. The package is actually installed, but in another Python venv, and not in the current one. It's of course hard to find putative "other" venvs on your computer, but you can look in obvious places like a venv/ dir in the project directory. pip list will show the packages installed in the current environment, so that can help identify them. You can of course just install them again in the current environment as an alternative to looking for the other one.
The environment stuff often comes up due to using "helper" programs, like conda or pyenv. IDEs with integrated Python consoles, like VS Code's Python tab, can also start it with a different environment than your terminal. It's useful to be mindful of which helper programs you are using and how they handle environments, and avoid helpers that are hard to understand.
As a brute force solution, you can try searching for the package or module xyz on your computer: rg xyz / (rg is https://github.com/BurntSushi/ripgrep). Because of Python's import system and conventions, if there is an importable xyz, there is probably a file xyz.py somewhere. You can analyze its path to figure out what environment it is.
0 comment threads
0 comment threads