Python does not explicitly provide begin and end nesting signals. Rather, the indentation level indicates when you begin and end. This is an interesting lexical problem because multiple DEDENT tokens should be sent to the parser sometimes without a corresponding input symbol! Consider the following example:

a=1
 if a>1:
     print a
 b=3

Here the "b" token on the left edge signals that a DEDENT is needed after the "print a \n" and before the "b". The sequence should be

 ... 1 COLON NEWLINE INDENT PRINT a NEWLINE DEDENT b ASSIGN 3 ...

For more examples, see the big comment at the bottom of this file. This TokenStream normally just passes tokens through to the parser. Upon NEWLINE token from the lexer, however, an INDENT or DEDENT token may need to be sent to the parser. The NEWLINE is the trigger for this class to do it's job. NEWLINE is saved and then the first token of the next line is examined. If non-leading-whitespace token, then check against stack for indent vs dedent. If LEADING_WS, then the column of the next non-whitespace token will dictate indent vs dedent. The column of the next real token is number of spaces in the LEADING_WS token + 1 (to move past the whitespace). The lexer grammar must set the text of the LEADING_WS token to be the proper number of spaces (and do tab conversion etc...). A stack of column numbers is tracked and used to detect changes in indent level from one token to the next. A queue of tokens is built up to hold multiple DEDENT tokens that are generated. Before asking the lexer for another token via nextToken(), the queue is flushed first one token at a time. Terence Parr and Loring Craymer February 2004