tokenize.detect_encoding() never checks what readline hands back, so a
readline returning str fails on whatever touches the line first. In the
common case that is first.startswith(BOM_UTF8), reporting "startswith
first arg must be str or a tuple of str, not bytes"; if the first line is
blank the failure moves into find_cookie() and becomes "cannot use a
bytes pattern on a string-like object". Neither mentions readline. Check
the line in read_or_stop() instead, and say both that readline has to
return bytes and where to go next: read the source as bytes, or use
generate_tokens() for text.
The exception type is unchanged, since str input raised TypeError before
and still does. The check rejects str rather than requiring bytes, so a
readline returning bytearray keeps working, and it sits after the
StopIteration handler so a readline that is already exhausted still
gives ('utf-8', []).
Closes #67486.
Passing a text readline to
tokenize.detect_encodingfails inside the BOM check, so you get an error aboutstartswithrather than about what you did wrong:>>> tokenize.detect_encoding(io.StringIO("x=1\n").readline) TypeError: startswith first arg must be str or a tuple of str, not bytes@berkerpeksag pointed out in 2019 that the case still reachable is
tokenize.open()together withtokenize.tokenize(), sinceopen()returns a text stream. That is still how it behaves on main today, with his own example file:With this change both say:
The check lives in
read_or_stop()insidedetect_encoding, after theStopIterationhandler, so a readline that stops immediately still returns('utf-8', []). It rejectsstrrather than demandingbytes, sobytearraykeeps working — that path isn't documented, but it works today and it seemed wrong to break it in passing.The exception type doesn't change: it was
TypeErrorbefore and still is. The message namesdetect_encodingrather thantokenize, becausedetect_encodingis public and called directly fromtrace.py,idlelib/iomenu.pyandimportlib/_bootstrap_external.py— pointing attokenize()would name a function that isn't in the caller's traceback.Three new cases in
TestDetectEncoding: str on the first line, str on the second, and bytearray as a control that the check didn't tighten intoisinstance(line, bytes). Changing the check tonot isinstance(line, bytes)fails the suite, so the bytearray case is doing work../python.exe -m test test_tokenize test_trace test_inspect test_linecacheis 600 tests, all passing. No existing test asserted the old message, so nothing had to be adjusted.One line added to the
detect_encodingdocs, since the issue has carried adocslabel since 2019. main only; this improves an error message rather than fixing a crash, so I'm not proposing a backport.