The Tiger Book: 2 Lexical Analysis

The lexical analyzer (lexer) takes a stream of characters and produces a stream of names, keywords, and punctuation marks; it discards white space and comments between the tokens. Lexical tokens have types in any programming language, example: ID foo, x, n14 NUM 73, 0, 082 REAL 3.14, 10., 1e9, 0.5 COMMA , NOTEQ != LPAREN ( RPAREN ) Punctuation tokens such as IF , VOID , RETURN constructed from…