Fix Python 3 behavioral bugs in handlers and SecurityManager#34
Open
thegushi wants to merge 13 commits into
Open
Fix Python 3 behavioral bugs in handlers and SecurityManager#34thegushi wants to merge 13 commits into
thegushi wants to merge 13 commits into
Conversation
iconv -o is a GNU iconv extension not supported on BSD. Replace the subprocess call with Python's own open() encoding support, which is portable and removes the iconv dependency entirely.
Replace file -bi encoding detection with a Python-native UTF-8 open attempt. If the file opens cleanly as UTF-8, skip it; otherwise convert from the known locale encoding. No external tools needed.
Detects legacy SHA1 hex digest passwords that need upgrading to the PBKDF2 format introduced by hash_password().
sha_new() requires bytes in Python 3 but was receiving a str. Switch to hash_password() so new lists get PBKDF2 hashes from the start rather than legacy SHA1. Fixes jaredmauch#24.
'Hit enter to notify %(listname)s owner...' was never interpolated. Replace print()+readline() with input() which handles both the prompt and waiting for Enter in one call.
check_perms and Mailman/MTA/Postfix.py both had Python 2-style
print C_('...') % locals(), statements where the migration to Python 3
dropped the % locals() substitution, leaving literal %(varname)s in
error output. Also fix two bare print -> print() in Postfix.py.
BSD make does not have a wildcard function -- it treats the argument as a variable name with spaces, generating warnings. Use $(POFILES) in messages/ (already defined) and drop the dependency list in templates/ (stamp file is sufficient for a clean build).
distutils was removed in Python 3.12. The configure check was using
distutils.sysconfig solely to verify Python development headers exist.
Replace with the sysconfig stdlib module (available since Python 3.2)
using sysconfig.get_path('include') in place of get_python_inc().
CookHeaders: filter empty recolon/subject before joining to avoid leading space in subject prefix (e.g. ' [XTEST] foo' -> '[XTEST] foo'). Decorate: prefer the message's own charset over the list charset when building the decorated payload. Passing bytes to set_payload() with a utf-8 charset causes Python 3's email library to base64-encode ASCII content unnecessarily. Verifying encodability without assigning keeps the payload as str, which set_payload() handles without adding CTE. Utils.verify_password: handle legacy md5 binary digest (16 raw bytes) stored before the SHA1 upgrade path existed. The function was rejecting these because they can't be decoded as ASCII. test_handlers: English list charset is now utf-8 (per Defaults.py.in), so acknowledgement messages are utf-8-encoded. Update charset assertions and use get_payload(decode=True) for content comparison. Fix hold notification cookie extraction to decode base64/utf-8 payload first.
CookHeaders: also fix leading space in Header path (non-us-ascii cset). When old_style=True and recolon is empty, the Header was initialized with an empty first chunk causing a leading space. When old_style=False, skip appending recolon when it is empty. Decorate multipart path: same charset fix as non-multipart — prefer mcset over lcset for MIMEText parts to avoid base64-encoding ASCII content. test_security_mgr: md5->PBKDF2 upgrade now works; update test to check for PBKDF2 format instead of the old SHA1 hexdigest intermediate format, and verify the upgraded password still authenticates.
test_multipart: Python 3's email generator adds Content-Transfer-Encoding: 7bit to multipart outer message and plain sub-messages; update expected string to match actual Python 3.11 output. Utils.verify_password: add legacy crypt(3) as a final fallback format so crypt-stored passwords can still authenticate and trigger the PBKDF2 upgrade path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Several Python 3 behavioral regressions in handlers, exposed by the test suite once the test infrastructure issues were cleaned up.
CookHeaders — leading space in subject prefix:
us-asciipath,' '.join([recolon, prefix, subject])prepended a space whenrecolonwas empty. Filter empty strings before joining.cset != 'us-ascii', e.g. English now uses utf-8),old_style=Trueinitialized the Header with an emptyrecolonas the first chunk, causing the same leading-space bug. Fix bothold_stylesub-cases to skip empty chunks.Decorate — base64-encoding ASCII content:
set_payload(bytes, 'utf-8'), triggering base64. Fix: verify encodability without converting to bytes; pass str toset_payload.MIMEText(text, 'plain', lcset)wherelcset='utf-8', causing the same issue. Fix: prefer the message's own charset for ASCII content.mcset) over the list charset (lcset), falling back tolcsetonly whenmcsetcan't encode the decorated content.SecurityManager — md5 binary digest not recognized:
verify_passwordcouldn't handle the legacy md5 binary digest format (16 raw bytes), returningFalse, Falsebecause the bytes couldn't be decoded as ASCII. Add an explicit check for 16-byte binary digests before the ASCII decode attempt.Tests:
Defaults.py.in); updatetest_ack_*charset assertions and useget_payload(decode=True)for payload comparison.test_list_admin_upgrade: md5 passwords now upgrade to PBKDF2 (not SHA1); update assertions accordingly.