CS 331 Spring 2009
Additional Lecture Notes for Wednesday, February 4, 2009

The following material was covered in class, in addition to the material from the text.

Writing a Lexical Analyzer

We began writing a lexical analyzer to analyze according to the scheme on the handout.

Some design decisions.

• Base the lexical analyzer on a state machine.
• A C++ class Lex has a constructor that takes the entire program as a parameter. Member function next returns the next lexeme. Member function done tells whether all lexemes have been returned.
• An object keeps a pointer to the beginning of the next lexeme, or just past the end of the string if there are no more lexemes. (This means that the constructor must do whitespace skipping.)

States needed, after skipping of whitespace and comments is handled.

• Start.
• States that can be entered after reading a single character:
  • Read digit.
  • Read letter or underscore.
  • Read “+”.
  • Read “-”.
  • Read “.”.
  • Read other operator.
  • Read illegal character.
• Other states:
  • Read digit followed by decimal-point.

States that result in returning a lexeme do not need to be listed; the lexeme can simply be returned (and thus the states “Read other operator&rdquo and “Read illegal character” may be unnecessary). Other combinations of characters will result in one of the above states being entered. For example, reading “+3” sends us to the “Digit” state, while reading “.9” sends us to the “Read digit followed by decimal-point” state.

CS 331 Spring 2009: Additional Lecture Notes for Wednesday, February 4, 2009 / Updated: 4 Feb 2009 / Glenn G. Chappell / ffggc@uaf.edu