Fix body_line_iterator(decode=True) returning no lines in Python 3.11#31
Open
thegushi wants to merge 13 commits into
Open
Fix body_line_iterator(decode=True) returning no lines in Python 3.11#31thegushi wants to merge 13 commits into
thegushi wants to merge 13 commits into
Conversation
iconv -o is a GNU iconv extension not supported on BSD. Replace the subprocess call with Python's own open() encoding support, which is portable and removes the iconv dependency entirely.
Replace file -bi encoding detection with a Python-native UTF-8 open attempt. If the file opens cleanly as UTF-8, skip it; otherwise convert from the known locale encoding. No external tools needed.
Detects legacy SHA1 hex digest passwords that need upgrading to the PBKDF2 format introduced by hash_password().
sha_new() requires bytes in Python 3 but was receiving a str. Switch to hash_password() so new lists get PBKDF2 hashes from the start rather than legacy SHA1. Fixes jaredmauch#24.
'Hit enter to notify %(listname)s owner...' was never interpolated. Replace print()+readline() with input() which handles both the prompt and waiting for Enter in one call.
check_perms and Mailman/MTA/Postfix.py both had Python 2-style
print C_('...') % locals(), statements where the migration to Python 3
dropped the % locals() substitution, leaving literal %(varname)s in
error output. Also fix two bare print -> print() in Postfix.py.
BSD make does not have a wildcard function -- it treats the argument as a variable name with spaces, generating warnings. Use $(POFILES) in messages/ (already defined) and drop the dependency list in templates/ (stamp file is sufficient for a clean build).
distutils was removed in Python 3.12. The configure check was using
distutils.sysconfig solely to verify Python development headers exist.
Replace with the sysconfig stdlib module (available since Python 3.2)
using sysconfig.get_path('include') in place of get_python_inc().
Python 3.11's body_line_iterator only yields str payloads. When decode=True, get_payload() returns bytes, which the iterator silently drops — leaving SimpleMatch, SimpleWarning, and Tagger with nothing to scan. Remove decode=True (defaulting to False) so get_payload() returns str as expected. Bounce detection and topic tagging now work correctly under Python 3.
The previous fix (removing decode=True) stopped the crash but left quoted-printable and base64 bodies undecoded, causing e.g. QP soft line breaks (=20) to appear literally in addresses and break pattern matching. Add _body_line_iterator() to SimpleMatch which uses get_payload(decode=True) to get CTE-decoded bytes, then decodes to str using the part's charset. SimpleWarning imports and uses the same helper. Tagger likewise imports it so topic matching works correctly on encoded message bodies.
In Python 3, get_payload(decode=True) always returns bytes. AOL.py was iterating over those bytes with string regex patterns, causing TypeError. Decode to str before iterating.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python 3.11's
email.iterators.body_line_iteratoronly yieldsstrpayloads. When called withdecode=True,get_payload()returnsbytes, which the iterator silently drops — leavingSimpleMatch,SimpleWarning, andTaggerwith an empty line set and therefore finding nothing.Removing
decode=True(reverting to the defaultFalse) causesget_payload()to returnstr, which the iterator handles correctly. Bounce detection and topic tagging now work under Python 3.Affected:
Mailman/Bouncers/SimpleMatch.pyMailman/Bouncers/SimpleWarning.pyMailman/Handlers/Tagger.pyThis was caught by the test suite (
test_bounces.BounceTest.test_bouncefailing for the SimpleMatch/sendmail case, and the Tagger tests failing intest_handlers).