loic-simon · GitHub

Bug report

Bug description:

Found when investigating test.test_tokenize.TestRoundtrip.test_random_files failing on #134577

def test(code):
    tokens = list(tokenize.tokenize(iter([code]).__next__))
    from5 = tokenize.untokenize(tokens)
    print("from5   ", from5, eval(from5))
    tokens2 = [tok[:2] for tok in tokens]
    from2 = tokenize.untokenize(tokens2)
    print("from2   ", from2, eval(from2))
>>> test(b't"{ {} }"')
from5    b't"{ {} }"' Template(strings=('', ''), interpolations=(Interpolation({}, ' {}', None, ''),))
from2    b't"{{}}"'   Template(strings=('{}',), interpolations=())
>>> test(b'f"{ {} }"')
from5    b'f"{ {} }"' {}
from2    b'f"{ {} }"' {}

From what I understand, untokenize should round-trip correctly even with only the type and string of tokens.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

Read the original on github.com ↗