Bug report
Bug description:
Found when investigating test.test_tokenize.TestRoundtrip.test_random_files failing on #134577
def test(code): tokens = list(tokenize.tokenize(iter([code]).__next__)) from5 = tokenize.untokenize(tokens) print("from5 ", from5, eval(from5)) tokens2 = [tok[:2] for tok in tokens] from2 = tokenize.untokenize(tokens2) print("from2 ", from2, eval(from2))
>>> test(b't"{ {} }"') from5 b't"{ {} }"' Template(strings=('', ''), interpolations=(Interpolation({}, ' {}', None, ''),)) from2 b't"{{}}"' Template(strings=('{}',), interpolations=()) >>> test(b'f"{ {} }"') from5 b'f"{ {} }"' {} from2 b'f"{ {} }"' {}
From what I understand, untokenize should round-trip correctly even with only the type and string of tokens.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS