pkgutil.walk_packages "prefix" option docs are misleading

[cykerway]

BPO	33210
Nosy	@ncoghlan, @ericsnowcurrently, @cykerway
Files	test.py: Test program showing difference of behavior.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-04-03.02:25:38.194>
labels = ['type-bug', '3.7', 'docs']
title = 'pkgutil.walk_packages "prefix" option docs are misleading'
updated_at = <Date 2018-04-06.14:30:40.552>
user = 'https://github.com/cykerway'

bugs.python.org fields:

activity = <Date 2018-04-06.14:30:40.552>
actor = 'ncoghlan'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2018-04-03.02:25:38.194>
creator = 'cykerway'
dependencies = []
files = ['47519']
hgrepos = []
issue_num = 33210
keywords = []
message_count = 4.0
messages = ['314850', '314960', '315017', '315018']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'docs@python', 'eric.snow', 'cykerway']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33210'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

• gh-77391: Say what walk_packages' prefix does to the imports #158138

[cykerway]

The current implementation of pkgutil.walk_packages() is confusing. Users may be given incomplete results while no exception is raised if they don't explicitly provide the prefix parameter. The doc says:

prefix is a string to output on the front of every module name on output.

But the fact is, prefix is not merely an output formatter at all. This function cannot handle an empty prefix (which is the default) and will often encounter import errors which are then simply ignored without an onerror function (which is default again).

See test program for details.

Doc:

https://docs.python.org/3.6/library/pkgutil.html#pkgutil.walk_packages

Source:

https://github.com/python/cpython/blob/3.6/Lib/pkgutil.py

[cykerway]

Update test program.

[ncoghlan]

I think this is actually two distinct problems, one documentation one (which should be addressed in the online docs for all currently maintained versions), and one actual functional issue.

The documentation issue is the one you've reported: in order for the recursive descent to work in walk_packages given the current algorithm, then the combination of the given prefix, and the current global import configuration must allow that package to actually be imported. While there is a note about that limitation, it's currently thoroughly unclear.

The functional issue is two-fold:

1. pkgutil.iter_modules() doesn't identify PEP-420 namespace packages correctly (it ignores them as not being potential packages)

2. The recursive import to check pkg.__path__ uses a name based global __import__ rather than the more state independent https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly technique

It's that second problem that introduces the "prefix must be set to get useful output" behaviour that you're currently seeing.

[ncoghlan]

https://bugs.python.org/issue29258 is an existing issue for the PEP-420 limitation (it also notes why fixing that implicitly is a potential problem)

I've retitled this issue to be specifically about the misleading docs for the "prefix" option (and updated the metadata accordingly).

"Recursive descent in pkgutil.walk_packages depends on sys.path" would be a reasonable title for an issue to actually fix the operation to use a better algorithm based on newer importlib APIs, but I haven't filed that myself.