The unknown-8bit trick was designed to deal with unknown bytes in an ASCII message, and it works fine for that. However, I also tried to extend it to handle bytes that can't be decoded using the charset specified in an encoded word, and there it fails because there can be other non-ASCII characters that were *successfully* decoded. The fix is simple: do the unknown-8bit encoding using the utf-8 codec. This is especially appropriate since anyone trying to do recovery on an unknown byte string will probably attempt utf-8 first.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request
Dec 24, 2025…-142517) The unknown-8bit trick was designed to deal with unknown bytes in an ASCII message, and it works fine for that. However, I also tried to extend it to handle bytes that can't be decoded using the charset specified in an encoded word, and there it fails because there can be other non-ASCII characters that were *successfully* decoded. The fix is simple: do the unknown-8bit encoding using the utf-8 codec. This is especially appropriate since anyone trying to do recovery on an unknown byte string will probably attempt utf-8 first. (cherry picked from commit 1e17ccd ) Co-authored-by: R. David Murray <rdmurray@bitdance.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request
Dec 24, 2025…-142517) The unknown-8bit trick was designed to deal with unknown bytes in an ASCII message, and it works fine for that. However, I also tried to extend it to handle bytes that can't be decoded using the charset specified in an encoded word, and there it fails because there can be other non-ASCII characters that were *successfully* decoded. The fix is simple: do the unknown-8bit encoding using the utf-8 codec. This is especially appropriate since anyone trying to do recovery on an unknown byte string will probably attempt utf-8 first. (cherry picked from commit 1e17ccd ) Co-authored-by: R. David Murray <rdmurray@bitdance.com>
bitdancer added a commit that referenced this pull request
Dec 24, 2025…H-142517) (#143147) The unknown-8bit trick was designed to deal with unknown bytes in an ASCII message, and it works fine for that. However, I also tried to extend it to handle bytes that can't be decoded using the charset specified in an encoded word, and there it fails because there can be other non-ASCII characters that were *successfully* decoded. The fix is simple: do the unknown-8bit encoding using the utf-8 codec. This is especially appropriate since anyone trying to do recovery on an unknown byte string will probably attempt utf-8 first. (cherry picked from commit 1e17ccd ) Co-authored-by: R. David Murray <rdmurray@bitdance.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
bitdancer added a commit that referenced this pull request
Dec 24, 2025…H-142517) (#143146) The unknown-8bit trick was designed to deal with unknown bytes in an ASCII message, and it works fine for that. However, I also tried to extend it to handle bytes that can't be decoded using the charset specified in an encoded word, and there it fails because there can be other non-ASCII characters that were *successfully* decoded. The fix is simple: do the unknown-8bit encoding using the utf-8 codec. This is especially appropriate since anyone trying to do recovery on an unknown byte string will probably attempt utf-8 first. (cherry picked from commit 1e17ccd ) Co-authored-by: R. David Murray <rdmurray@bitdance.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Closed
Merged