The first stage of processing a source file is the division of the source file into lexical elements.
A lexical element is one of the following:
- a maximal sequence of whitespace characters
- a word
- a literal:
- TODO: operators, comments, ...
The sequence of lexical elements is formed by repeatedly removing the longest initial sequence of characters that forms a valid lexical element.