Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Community Proposals
Community Proposals
tag:snake search within a tag
answers:0 unanswered questions
user:xxxx search by author id
score:0.5 posts with 0.5+ score
"snake oil" exact phrase
votes:4 posts with 4+ votes
created:<1w created < 1 week ago
post_type:xxxx type of post
Search help
Notifications
Mark all as read See all your notifications »
Q&A

Welcome to Software Development on Codidact!

Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.

Post History

80%
+6 −0
Q&A How can I properly type-hint methods in different files that would lead to circular imports?

Import modules rather than names first to avoid a circular reference in the import statements; then use forward declarations, as before, to avoid a circular reference in the type annotations - like...

posted 9mo ago by Karl Knechtel‭  ·  edited 9mo ago by Karl Knechtel‭

Answer
#3: Post edited by user avatar Karl Knechtel‭ · 2023-08-21T14:16:53Z (9 months ago)
mention specific point when the error-message improvement was added
  • **Import modules rather than names first** to avoid a circular reference in the `import` statements; **then use forward declarations**, as before, to avoid a circular reference in the type annotations - like so:
  • ```
  • # process.py
  • import helpers
  • class Process:
  • def do_something(self, helper: "helpers.Helper"):
  • ...
  • ```
  • ```
  • # helpers.py
  • import process
  • class Helper:
  • def update(self, process: "process.Process"):
  • ...
  • ```
  • Circular imports are entirely possible in Python; problems only occur due to circular references in the imported code **if and when that code runs at import time**. It's important to keep in mind here that `class` and `def` **are statements** in Python, so when they appear at top level they represent code that will run immediately. (The effect of "running" `def` is to *create the object representing the function* and assign it to the corresponding name; similarly for classes.)
  • <details>
  • <summary>Background information about the import process</summary>
  • When Python evaluates an `import` statement (At least, by default - the import system provides a *ton* of hooks to customize the behaviour), once it has determined that the module actually needs to be loaded (and that there is a file to load it from, and where that file is) it first creates an empty `module` object and stores it with the appropriate name in a global module dict (available by default as `sys.modules`) - the same one that it uses to check whether a module has already been loaded. It then evaluates the top-level code of that file, *using the attributes of the `module` object* as the global namespace for that code.
  • This has a few important implications:
  • * `import` is, of course, also a statement, and thus any top-level `import`s in the module being imported will follow the same logic, recursively. However, because an object was stored in `sys.modules` *before* executing the module code, an ordinary loop of `import` statements doesn't cause a problem by itself. If we `import process` in the above example, it will `import helpers`, which will `import process` - which will find the empty module object in `sys.modules`, and therefore *not* attempt to locate `process.py` again. (As a historical note, this didn't always work properly in all cases: see https://github.com/python/cpython/issues/74210 and https://github.com/python/cpython/issues/61836 .)
  • * However, a problem emerges with circular imports when the *top-level code depends on the contents of* the imported module. If we `import process`, such that `helpers` finds an empty `process` module in the `sys.modules` lookup, *it will not be able to use* any attributes of `process` that haven't yet been assigned. Python automatically converts the resulting `AttributeError` into an `ImportError` internally. In more recent versions, it can inspect the stack to add `(most likely due to a circular import)` to the exception message, as opposed to `ImportError`s caused by modules that simply don't define the name in question.
  • * Normally, we put `import` statements at the top of the code to avoid confusion. But they are just ordinary statements that can run at any time. In particular, it's possible to define global variables in `process.py` *before* the `import helpers` line, and then have `helpers.py` import them `from process`.
  • * It's also possible to have an `import` inside the code of a function, which will then not be attempted until the function is called. (It will be attempted every time the function is called, **but** the cached module ordinarily will be immediately found every time after the first.) However, that doesn't help with the current case, because both the `class` and `def` statements will run immediately.
  • * It's possible for the top-level code to *replace* the object stored in `sys.modules` - for example, by defining a `_Implementation` class and then assigning `sys.modules[__name__] = _Implementation()`. In this case, the global names from the module *still get attached to the original `module` object, not* whatever was stored into `sys.modules` (i.e. they won't interfere with the class instance; the class code can still use those names, because the module object is still acting as the class' global namespace). This can be used to get the effect of modules that seem to have dynamically determined "magic" attributes (by implementing `__getattr__` or `__getattribute__` in the class).
  • </details>
  • Because of the `import`s, MyPy should now have enough information to resolve the string forward references and understand what type is being named. Meanwhile, since the forward reference is only a string, it doesn't cause a complaint from Python at runtime: when Python constructs the `__annotations__` for the `update` and `do_something` functions, it just stores those strings rather than having to look up any other names.
  • **Import modules rather than names first** to avoid a circular reference in the `import` statements; **then use forward declarations**, as before, to avoid a circular reference in the type annotations - like so:
  • ```
  • # process.py
  • import helpers
  • class Process:
  • def do_something(self, helper: "helpers.Helper"):
  • ...
  • ```
  • ```
  • # helpers.py
  • import process
  • class Helper:
  • def update(self, process: "process.Process"):
  • ...
  • ```
  • Circular imports are entirely possible in Python; problems only occur due to circular references in the imported code **if and when that code runs at import time**. It's important to keep in mind here that `class` and `def` **are statements** in Python, so when they appear at top level they represent code that will run immediately. (The effect of "running" `def` is to *create the object representing the function* and assign it to the corresponding name; similarly for classes.)
  • <details>
  • <summary>Background information about the import process</summary>
  • When Python evaluates an `import` statement (At least, by default - the import system provides a *ton* of hooks to customize the behaviour), once it has determined that the module actually needs to be loaded (and that there is a file to load it from, and where that file is) it first creates an empty `module` object and stores it with the appropriate name in a global module dict (available by default as `sys.modules`) - the same one that it uses to check whether a module has already been loaded. It then evaluates the top-level code of that file, *using the attributes of the `module` object* as the global namespace for that code.
  • This has a few important implications:
  • * `import` is, of course, also a statement, and thus any top-level `import`s in the module being imported will follow the same logic, recursively. However, because an object was stored in `sys.modules` *before* executing the module code, an ordinary loop of `import` statements doesn't cause a problem by itself. If we `import process` in the above example, it will `import helpers`, which will `import process` - which will find the empty module object in `sys.modules`, and therefore *not* attempt to locate `process.py` again. (As a historical note, this didn't always work properly in all cases: see https://github.com/python/cpython/issues/74210 and https://github.com/python/cpython/issues/61836 .)
  • * However, a problem emerges with circular imports when the *top-level code depends on the contents of* the imported module. If we `import process`, such that `helpers` finds an empty `process` module in the `sys.modules` lookup, *it will not be able to use* any attributes of `process` that haven't yet been assigned. Python automatically converts the resulting `AttributeError` into an `ImportError` internally. Since 3.8, Python can inspect the stack to add `(most likely due to a circular import)` to the exception message, as opposed to `ImportError`s caused by modules that simply don't define the name in question. (I couldn't find a reference to this in the documentation, but I confirmed it manually by comparing `ceval.c` in the source across versions.)
  • * Normally, we put `import` statements at the top of the code to avoid confusion. But they are just ordinary statements that can run at any time. In particular, it's possible to define global variables in `process.py` *before* the `import helpers` line, and then have `helpers.py` import them `from process`.
  • * It's also possible to have an `import` inside the code of a function, which will then not be attempted until the function is called. (It will be attempted every time the function is called, **but** the cached module ordinarily will be immediately found every time after the first.) However, that doesn't help with the current case, because both the `class` and `def` statements will run immediately.
  • * It's possible for the top-level code to *replace* the object stored in `sys.modules` - for example, by defining a `_Implementation` class and then assigning `sys.modules[__name__] = _Implementation()`. In this case, the global names from the module *still get attached to the original `module` object, not* whatever was stored into `sys.modules` (i.e. they won't interfere with the class instance; the class code can still use those names, because the module object is still acting as the class' global namespace). This can be used to get the effect of modules that seem to have dynamically determined "magic" attributes (by implementing `__getattr__` or `__getattribute__` in the class).
  • </details>
  • Because of the `import`s, MyPy should now have enough information to resolve the string forward references and understand what type is being named. Meanwhile, since the forward reference is only a string, it doesn't cause a complaint from Python at runtime: when Python constructs the `__annotations__` for the `update` and `do_something` functions, it just stores those strings rather than having to look up any other names.
#2: Post edited by user avatar Karl Knechtel‭ · 2023-08-21T14:06:37Z (9 months ago)
  • **Import modules rather than names first** to avoid a circular reference in the `import` statements, and **then use forward declarations**, as before, to avoid a circular reference in the type annotations. Like so:
  • ```
  • # process.py
  • import helpers
  • class Process:
  • def do_something(self, helper: "helpers.Helper"):
  • ...
  • ```
  • ```
  • # helpers.py
  • import process
  • class Helper:
  • def update(self, process: "process.Process"):
  • ...
  • ```
  • Circular imports are entirely possible in Python; problems only occur due to circular references in the imported code **if and when that code runs at import time**. It's important to keep in mind here that `class` and `def` **are statements** in Python, so when they appear at top level they represent code that will run immediately. (The effect of "running" `def` is to *create the object representing the function* and assign it to the corresponding name; similarly for classes.)
  • <details>
  • <summary>Background information about the import process</summary>
  • When Python evaluates an `import` statement[^1], once it has determined that the module actually needs to be loaded (and that there is a file to load it from, and where that file is) it first creates an empty `module` object and stores it with the appropriate name in a global module dict (available by default as `sys.modules`) - the same one that it uses to check whether a module has already been loaded. It then evaluates the top-level code of that file, *using the attributes of the `module` object* as the global namespace for that code.
  • This has a few important implications:
  • * `import` is, of course, also a statement, and thus any top-level `import`s in the module being imported will follow the same logic, recursively. However, because an object was stored in `sys.modules` *before* executing the module code, an ordinary loop of `import` statements doesn't cause a problem by itself. If we `import process` in the above example, it will `import helpers`, which will `import process` - which will find the empty module object in `sys.modules`, and therefore *not* attempt to locate `process.py` again. (As a historical note, this didn't always work properly in all cases: see https://github.com/python/cpython/issues/74210 and https://github.com/python/cpython/issues/61836 .)
  • * However, a problem emerges with circular imports when the *top-level code depends on the contents of* the imported module. If we `import process`, such that `helpers` finds an empty `process` module in the `sys.modules` lookup, *it will not be able to use* any attributes of `process` that haven't yet been assigned. Python automatically converts the resulting `AttributeError` into an `ImportError` internally. In more recent versions, it can inspect the stack to add `(most likely due to a circular import)` to the exception message, as opposed to `ImportError`s caused by modules that simply don't define the name in question.
  • * Normally, we put `import` statements at the top of the code to avoid confusion. But they are just ordinary statements that can run at any time. In particular, it's possible to define global variables in `process.py` *before* the `import helpers` line, and then have `helpers.py` import them `from process`.
  • * It's also possible to have an `import` inside the code of a function, which will then not be attempted until the function is called. (It will be attempted every time the function is called, **but** the cached module ordinarily will be immediately found every time after the first.) However, that doesn't help with the current case, because both the `class` and `def` statements will run immediately.
  • * It's possible for the top-level code to *replace* the object stored in `sys.modules` - for example, by defining a `_Implementation` class and then assigning `sys.modules[__name__] = _Implementation()`. In this case, the global names from the module *still get attached to the original `module` object, not* whatever was stored into `sys.modules` (i.e. they won't interfere with the class instance; the class code can still use those names, because the module object is still acting as the class' global namespace). This can be used to get the effect of modules that seem to have dynamically determined "magic" attributes (by implementing `__getattr__` or `__getattribute__` in the class).
  • </details>
  • Because of the `import`s, MyPy should now have enough information to resolve the string forward references and understand what type is being named. Meanwhile, since the forward reference is only a string, it doesn't cause a complaint from Python at runtime: when Python constructs the `__annotations__` for the `update` and `do_something` functions, it just stores those strings rather than having to look up any other names.
  • [^1]: At least, by default - the import system provides a *ton* of hooks to customize the behaviour.
  • **Import modules rather than names first** to avoid a circular reference in the `import` statements; **then use forward declarations**, as before, to avoid a circular reference in the type annotations - like so:
  • ```
  • # process.py
  • import helpers
  • class Process:
  • def do_something(self, helper: "helpers.Helper"):
  • ...
  • ```
  • ```
  • # helpers.py
  • import process
  • class Helper:
  • def update(self, process: "process.Process"):
  • ...
  • ```
  • Circular imports are entirely possible in Python; problems only occur due to circular references in the imported code **if and when that code runs at import time**. It's important to keep in mind here that `class` and `def` **are statements** in Python, so when they appear at top level they represent code that will run immediately. (The effect of "running" `def` is to *create the object representing the function* and assign it to the corresponding name; similarly for classes.)
  • <details>
  • <summary>Background information about the import process</summary>
  • When Python evaluates an `import` statement (At least, by default - the import system provides a *ton* of hooks to customize the behaviour), once it has determined that the module actually needs to be loaded (and that there is a file to load it from, and where that file is) it first creates an empty `module` object and stores it with the appropriate name in a global module dict (available by default as `sys.modules`) - the same one that it uses to check whether a module has already been loaded. It then evaluates the top-level code of that file, *using the attributes of the `module` object* as the global namespace for that code.
  • This has a few important implications:
  • * `import` is, of course, also a statement, and thus any top-level `import`s in the module being imported will follow the same logic, recursively. However, because an object was stored in `sys.modules` *before* executing the module code, an ordinary loop of `import` statements doesn't cause a problem by itself. If we `import process` in the above example, it will `import helpers`, which will `import process` - which will find the empty module object in `sys.modules`, and therefore *not* attempt to locate `process.py` again. (As a historical note, this didn't always work properly in all cases: see https://github.com/python/cpython/issues/74210 and https://github.com/python/cpython/issues/61836 .)
  • * However, a problem emerges with circular imports when the *top-level code depends on the contents of* the imported module. If we `import process`, such that `helpers` finds an empty `process` module in the `sys.modules` lookup, *it will not be able to use* any attributes of `process` that haven't yet been assigned. Python automatically converts the resulting `AttributeError` into an `ImportError` internally. In more recent versions, it can inspect the stack to add `(most likely due to a circular import)` to the exception message, as opposed to `ImportError`s caused by modules that simply don't define the name in question.
  • * Normally, we put `import` statements at the top of the code to avoid confusion. But they are just ordinary statements that can run at any time. In particular, it's possible to define global variables in `process.py` *before* the `import helpers` line, and then have `helpers.py` import them `from process`.
  • * It's also possible to have an `import` inside the code of a function, which will then not be attempted until the function is called. (It will be attempted every time the function is called, **but** the cached module ordinarily will be immediately found every time after the first.) However, that doesn't help with the current case, because both the `class` and `def` statements will run immediately.
  • * It's possible for the top-level code to *replace* the object stored in `sys.modules` - for example, by defining a `_Implementation` class and then assigning `sys.modules[__name__] = _Implementation()`. In this case, the global names from the module *still get attached to the original `module` object, not* whatever was stored into `sys.modules` (i.e. they won't interfere with the class instance; the class code can still use those names, because the module object is still acting as the class' global namespace). This can be used to get the effect of modules that seem to have dynamically determined "magic" attributes (by implementing `__getattr__` or `__getattribute__` in the class).
  • </details>
  • Because of the `import`s, MyPy should now have enough information to resolve the string forward references and understand what type is being named. Meanwhile, since the forward reference is only a string, it doesn't cause a complaint from Python at runtime: when Python constructs the `__annotations__` for the `update` and `do_something` functions, it just stores those strings rather than having to look up any other names.
#1: Initial revision by user avatar Karl Knechtel‭ · 2023-08-21T14:05:49Z (9 months ago)
**Import modules rather than names first** to avoid a circular reference in the `import` statements, and **then use forward declarations**, as before, to avoid a circular reference in the type annotations. Like so:

```
# process.py

import helpers

class Process:
    def do_something(self, helper: "helpers.Helper"):
        ...
```
```
# helpers.py

import process

class Helper:
    def update(self, process: "process.Process"):
        ...
```
Circular imports are entirely possible in Python; problems only occur due to circular references in the imported code **if and when that code runs at import time**. It's important to keep in mind here that `class` and `def` **are statements** in Python, so when they appear at top level they represent code that will run immediately. (The effect of "running" `def` is to *create the object representing the function* and assign it to the corresponding name; similarly for classes.)

<details>
<summary>Background information about the import process</summary>

When Python evaluates an `import` statement[^1], once it has determined that the module actually needs to be loaded (and that there is a file to load it from, and where that file is) it first creates an empty `module` object and stores it with the appropriate name in a global module dict (available by default as `sys.modules`) - the same one that it uses to check whether a module has already been loaded. It then evaluates the top-level code of that file, *using the attributes of the `module` object* as the global namespace for that code.

This has a few important implications:

* `import` is, of course, also a statement, and thus any top-level `import`s in the module being imported will follow the same logic, recursively. However, because an object was stored in `sys.modules` *before* executing the module code, an ordinary loop of `import` statements doesn't cause a problem by itself. If we `import process` in the above example, it will `import helpers`, which will `import process` - which will find the empty module object in `sys.modules`, and therefore *not* attempt to locate `process.py` again. (As a historical note, this didn't always work properly in all cases: see https://github.com/python/cpython/issues/74210 and https://github.com/python/cpython/issues/61836 .)

* However, a problem emerges with circular imports when the *top-level code depends on the contents of* the imported module. If we `import process`, such that `helpers` finds an empty `process` module in the `sys.modules` lookup, *it will not be able to use* any attributes of `process` that haven't yet been assigned. Python automatically converts the resulting `AttributeError` into an `ImportError` internally. In more recent versions, it can inspect the stack to add `(most likely due to a circular import)` to the exception message, as opposed to `ImportError`s caused by modules that simply don't define the name in question.

* Normally, we put `import` statements at the top of the code to avoid confusion. But they are just ordinary statements that can run at any time. In particular, it's possible to define global variables in `process.py` *before* the `import helpers` line, and then have `helpers.py` import them `from process`.

* It's also possible to have an `import` inside the code of a function, which will then not be attempted until the function is called. (It will be attempted every time the function is called, **but** the cached module ordinarily will be immediately found every time after the first.) However, that doesn't help with the current case, because both the `class` and `def` statements will run immediately.

* It's possible for the top-level code to *replace* the object stored in `sys.modules` - for example, by defining a `_Implementation` class and then assigning `sys.modules[__name__] = _Implementation()`. In this case, the global names from the module *still get attached to the original `module` object, not* whatever was stored into `sys.modules` (i.e. they won't interfere with the class instance; the class code can still use those names, because the module object is still acting as the class' global namespace). This can be used to get the effect of modules that seem to have dynamically determined "magic" attributes (by implementing `__getattr__` or `__getattribute__` in the class).
</details>

Because of the `import`s, MyPy should now have enough information to resolve the string forward references and understand what type is being named. Meanwhile, since the forward reference is only a string, it doesn't cause a complaint from Python at runtime: when Python constructs the `__annotations__` for the `update` and `do_something` functions, it just stores those strings rather than having to look up any other names.

[^1]: At least, by default - the import system provides a *ton* of hooks to customize the behaviour.